It’s a fact of digital life – outages and disruptions are going to happen. Whether they’re unavoidable natural disasters like Hurricane Sandy, security breaches like the one that affected Epsilon customers in 2011, or Responsys’ technology issues on Cyber Monday, there are and will always be things that knock you off course.
That is not to say that because outages are a fact of life they somehow aren’t bad, because they certainly are. Really bad, in fact, and here’s why: Service outages are never temporary. They have a ripple effect that extends well beyond the outage itself. For example, in Responsys’ case, where the outage occurred during the busiest online shopping day in history (and, for some, the highest email volume day of the year), the immediate impact was that retailers were unable to deploy the carefully crafted email campaigns they were counting on to drive a significant percentage of their Cyber Monday business. But the ripples will extend well beyond that impact:
- There were dollars invested creating those probably now-shelved campaigns that are now lost for some, and in those cases new dollars likely had to be invested altering creative and copy – not to mention more time and expense creating, testing, and deploying the new emails (and whether they could be made to have an impact in a short window is an open question).
- There were likely a number of triggered sends that didn’t happen – purchase confirmations, shipping information, abandoned cart, etc. – sends that consumers were expecting to see in their inboxes. Their customer service complaints won’t be directed at Responsys; the affected retailers will need to field those inquiries eating up resources that are needed elsewhere.
- Subscribers of affected retailers were left uninformed on products and cost-savings about which they had specifically requested to be informed by opting in. In those cases, many would likely spend their dollars elsewhere. These lost opportunities would be magnified if the retailer had planned a Cyber Monday-only price reduction or, worse, an in-store tie in.
- Assuming consumers either didn’t purchase or they purchased from another source, the affected retailers not only lost immediate revenue, but they would have long-term lost/lowered revenue on purchases associated with loyalty. You can’t suggest accessories or companion items for something that wasn’t purchased (but your unaffected competitors will if your customers purchased from them instead).
To this point we’ve established that outages are going to happen, and that when they do they’re bad. So what’s a retailer to do?
The only way to mitigate how bad an outage can be is by knowing what you’re going to do when the situation arises. In a crisis, you don’t want to spend precious hours meeting with stakeholders trying to figure out your next move. Smart companies take pains to understand what their vulnerabilities are and to have a ready and waiting playbook for when the unthinkable happens.
There are some basic principles that guide crisis planning, especially as it relates to issues of technology in general and platform selection in specific. First and foremost of these is this: Pick your partners wisely.
As you’re considering a platform partner, ask a lot of questions. Understand their security practices and the strategies that they will use to continually strengthen them. Explore their redundancies, their contingency planning, crisis support (especially their communications standards), and recovery procedures. Know your SLA (Service Level Agreement); it’s more than a contract, it’s a commitment to you and your business. How does your prospective partner expect to meet that commitment? It’s their problem, but it’s your reputation.
And while the outage might be on your partner, what you do during the outage is your responsibility. Your playbook is your commitment to your customers – how are you going to help them? Establish what your values are for a given outage type well in advance. In some situations a continuation of service to your clients to lessen the financial impact is appropriate, but in others it’s wholly inappropriate. Who are your internal and external teammates during the crisis? They probably vary depending on the nature of the outage. Do you even know how to get in touch with them during, say, a blizzard? Who communicates to your customers and how? Do you have a Twitter feed ready and waiting to use for communicating outage-related issues?
Take the time to establish your plan. Write it down – complete with interim responses (starting with an acknowledgement of the problem) and a plan for getting those responses out. Diligently socialize them with leadership and get buy-in (not just sign-off) from all of the players. Practice what needs to be practiced so that even if your teammates don’t know exactly what to do, they know where to go for answers. Last, make a plan to revisit and revise your playbook. Who knows how many long-established crisis playbooks there are out there that don’t have a plan addressing, say, a phishing scam under your brand’s masthead.
Let’s face facts: It’s going to happen. We can hope that when it does it’s a short outage and that it doesn’t affect anyone, but hope isn’t enough. Crises always have heroes; it’s your responsibility to take a few hours while the sun is still shining and plan how you’re going to save the day. Do the work to be prepared for the blizzard and then celebrate when it’s just a few flakes.