In my experience, disaster recovery (DR) and business continuity(BC) are either done very well or very poorly. There doesn't seemto be a middle ground. Most big organizations have redundant datacenters, replicated databases, failover servers, and a detailedbusiness continuity plan. Smaller organizations tend to have a fewpoint solutions for DR but do not (or are unable to) invest inalternate data centers and do not have failover solutions in place.If we think in terms of the insurance world, major carriers willhave effectively managed DR and BC plans and procedures. Agencies,GMAs, and regional carriers probably will struggle to get back ontheir feet after a real data disaster.

Social networking and the whole Web 2.0 phenomena have changedthe way we do many things. YouTube, Twitter, Facebook, andubiquitous Web-enabled smart phones now are part of everyinformation worker's life. Maybe we should embrace some of thesethings when we are planning for disaster recovery.

What We've Got Here Is Failure ToCommunicate

What is the most important thing we need when recovering fromdata disaster? Communications! Strother Martin said it all in CoolHand Luke. I need to be able to communicate with all my teammembers no matter where they are. Presumably some will be at thealternate center, others will be at the site of the disaster, andothers will be working from home or gathered at a site where theyhave Internet access. By the way, you need to have a couple ofsites scouted and prearranged if the corporate center or office isunavailable.

So, your team is dispersed, and it needs to interact. Everyonehas a cell phone, so they are able to speak with each other. Butyou know, sometimes an e-mail is 20 times more efficient than aphone conversation. Instant messaging also is effective fortechnical teams operating remotely. When I work remotely, I alwaysuse the corporate IM solution to communicate with fellow workers. Imay be termed onto a server from my home office using IM to tellthe techie in the data center what I need him to do. The problem isnow we are in disaster mode. The primary data center is down–wehave no corporate e-mail system, we have no corporate Internetconnection. The corporate phone system is voice over IP, so that isdown. The communications server is gone, so we have no IM. We areleft with only cell phones for communication–and they don't evenwork in the alternate data center.

Tweet Your DR?

With the data center going down, so do all communications. Allmembers of the team have to be called one at a time to inform themof the situation. Wouldn't it be more efficient to have a corporateDR team Twitter account? A quick tweet, and all members of the DRteam could be informed of the problem simultaneously–and byprearranged plan everyone could log onto a Web-based collaborationspace and an action plan put into process. There exist multipleno-cost solutions right now where dispersed team members could meetonline. Microsoft, Google, Yahoo, and others provide this type offunctionality for free.

There also are no-cost instant-messaging services. You could useSkype or something similar to create a quick online meeting usingvoice and video (Web cams or online white boards). Everyone on theDR team also should have a Google or Hotmail or Yahoo account. Allof these "free" communications tools are available using a smartphone. Everyone on that DR team also should have a data card. Sure,everyone has Internet access at home, but we want to have as manydifferent ways of getting online as possible. If your corporatedata center is down, there is a reasonable expectation the Wi-Fi atthe local Starbucks also might be down. All of this can bepreplanned and tested. I recommend creating an entire map ofpreconfigured communications alternatives.

It's Not Secure!

OK, I understand there may be resistance to using "public"hosted facilities to manage your disaster recovery–although I don'treally understand why that is an issue. I guarantee you will becommunicating using cellular communications that don't even need tobe hacked to be listened to. All you need to do is come up on theappropriate frequency, and all your conversations are there for thelistening. Logging onto a Gmail or Skype account at least providesa modicum of security.

On Cloud Nine

There is an alternative. And this is an alternative that shouldbe attractive to those smaller organizations that don't have theability to maintain multiple data centers. Cloud computing is hereto stay. It is a whole different beast than maintaining a hosteddata center off site. Cloud computing provides common businessapplications that are offered as online services by organizationswith large distributed infrastructures. Cloud services aredelivered by vendors such as Salesforce, Microsoft, Amazon, andothers. These firms have geographically distributed infrastructuresthat are virtually guaranteed to be available in all but majorglobal disasters.

Smaller organizations (and perhaps large ones, too) may not havethe staff and resources needed to manage common businessapplications as well as their line-of-business applications. If Iwere running an MGA, I would rather have my IT staff dedicated torevenue-producing applications and let someone else manage thingssuch as corporate e-mail, communications servers, CRM, intranets,Internet, and perhaps even productivity suites. Placing thesecommon business applications in the cloud eliminates licensingissues in addition to reducing the workload on IT staff. It alsoeliminates the need for a rack of servers that continually need tobe updated and maintained.

From the DR/BC perspective, the cloud becomes even moreattractive. Even in the teeth of a catastrophic failure of yourcorporate data center, you still will have access to and use ofcommon business applications. Your disaster recovery team will beable to devote its full energy to returning business applicationsto functionality. It will have an entire communications andcollaboration suite available–on the cloud. Furthermore, since weare concerned with business continuity only for a select group ofapplications, we can reduce the size of our alternate data center.The total cost of ownership for IT applications should be reduced.You can consume cloud services for fees that are generally lowerthan the TCO for those applications if they were hosted andmaintained internally.

Not My Data!

There currently is a significant amount of resistance toSoftware as a Service and cloud computing in general. Manyorganizations are uncomfortable relinquishing control of chunks oftheir corporate data to giants named Google, Microsoft, and Amazon.Search engines are intrusive. "Free" online Web analytics intrudeeven further. Placing confidential corporate data on the cloudsimply is too much for some organizations. I totally understandthis perspective, yet I do like the idea of some applications,particularly corporate e-mail and corporate messaging, provided asa service. Proper training and governance should ensure data thatshouldn't be publicly available under any circumstances be kept offthe cloud.

Test Time

If an organization does have a DR plan, chances are pretty goodit will periodically test its facilities and procedures. Most oftenDR tests are planned desktop exercises. The team is gatheredtogether, and the event is kicked off. The scenario is described,the data center is declared offline, certain team members aredesignated as incapacitated, and the plan goes into effect. Backupdata is requested and loaded onto the appropriate servers anddatabases in the alternate data center. On (very) rare occasions,an application actually is failed over to the DR server. Theexercise has been planned for weeks. The procedures were rehearsed,and servers were carefully checked for readiness before theexercise. So, what did the exercise really prove? That given twoweeks' notice, we probably can recover from a controlled datacenter problem?

A Real Test

When I was taking my check flight for a pilot's license, thecheck pilot reached over and killed the engine when I was justbarely airborne. I had to react immediately to lower the nose ofthe plane to maintain sufficient airspeed. At the same time, I hadto reverse the direction of the aircraft while calling the tower todeclare an emergency and clear the active runway from all othertraffic. I was coming back in dead stick and going the wrongdirection. Now, that was a real test of how I would react in anemergency. To be fair, I did expect the check pilot would pull somekind of funny business during the flight, but I did not expect itto be in the first 30 seconds. And I also am certain the checkpilot already had notified the tower he would bring us back inimmediately after takeoff. Nevertheless, that particular exerciseprovided some useful output. First, it "proved" to the check pilotI was reasonably capable of handling an unexpected problem, and itgave me a certain amount of confidence in my abilities.

Perhaps we should do something similar with our disasterrecovery training. Every so often, it makes sense to pull a systemoffline to see how your team reacts to an unscheduled emergency. Iam not suggesting you pull your ratings or policy admin systemoffline, but I am suggesting you periodically (once a year) andunannounced bring down some mission-critical internal system to seehow your team handles it. Hopefully, it will come through withflying colors, but if it doesn't, you will have pinpointed flaws inthe system. And that information is invaluable.

Keep an Open Mind

No matter where you come down right now in terms of preparednessfor a data disaster, you constantly need to evaluate the tools andoptions available. Don't be satisfied with a plan that made sensefive years ago. Consider alternative ways of doing things. As ithappens, my BlackBerry is down this weekend. I am communicatingusing my iPod touch–corporate e-mail, Gmail, Skype, IM, and Webbrowsing. I am not suggesting iPods as an alternative to some ofthings we discussed, but I am suggesting you consider other ways ofdoing things. Free doesn't mean bad. It just means the revenuestream is coming from a different source. And you should feel freeto look around and see whether some of the latest "cool"technologies might make sense in the corporate world. TD

Please address comments, complaints, and suggestions to theauthor at [email protected]