Richmond, Va., is a little over an hours drive from the coastalcities of Norfolk and Virginia Beach. Most people would considerRichmond far enough inland so that worries over hurricanes ortropical storms wouldnt cause a run on plywood at Home Depot. Thatis until Hurricane Isabel struck the Virginia coast in Septemberand pushed high winds all the way to Richmond. Jeff Fehn, directorof IT for Richmond-based Virginia Farm Bureau Insurance, figuredheavy rains would be the biggest problem for his city. At leastthats what happened during past hurricanes. But for some reasonIsabel was different. We got a pretty good brunt of it, hesays.

Fehn and other insurance IT leaders may be shocked when huge stormshit their area, or when an electrical blackout occurs, or even whena virus attacks their systems. They no longer can afford to besurprised, though. Too much is riding on the data that sits insidetheir data warehouse. These disasters serve as a warning toinsurers in other parts of the world, as well: You never know whentrouble will arise, so take some steps to prepare for theworst.

When you are talking about the blackout, I think a lot of peoplewere unprepared for a disaster that affected an entire region, saysMark Vanston, a program director for META Group. A good example isa lot of data centers have dual power feeds, so if one went down,they thought they were protected by a secondary source. The energyblackout that blanketed the Northeast section of the United Statesand parts of Canada on Aug. 16 made that strategy moot.

Power Is Out
In the aftermath of Isabel, power was out in Richmond for quitesome time, but Virginia Farm Bureau was ready to go with twodiesel-powered generators. The building had power, but not enoughto fuel the air conditioning for the next 10 days, except in thedata center. That was a minor inconvenience, Fehn says. The datacenter chillers ran just fine. That probably was the most popularspot in the building.

GHS Data Management is a claims processing outsourcer for pharmacyclaims. Located in Augusta, Maine, GHS escaped the blackout, butthe fierce New England winters have hardened businesses there tothe realities of natural disasters. In 1998, we had a huge icestorm, and there were people out of power for a few weeks, saysJason Skeffington, GHS data warehouse ad-ministrator. We have allour servers and mission-critical systems on a large UPS that issupplemented by a backup generator. If the power was wiped out inthis area for days or weeks, we still could operate, including ourdecision-support system, our data warehouse, and our Webserver.

What to Do
The typical steps taken when a data warehouse is designed is tobuild duplication or redundancy into the architecture, according toRon Barker, senior principal and insurance practice leader forKnightsbridge Solutions, a data warehouse and business intelligenceconsultancy. Companies will need a backup and disaster recoveryplan in place for each of their operational systems, which will besome strategy to back the data up nightly, weekly, monthly, andannualized backups, Barker says. They either will store the filesphysically off site in the event of a disaster or have a hotstandby operational data center, he says.
In the case of the big blackout, though, some insurers were relyingon third-party vendors to protect the data and found that wasnt agreat solution if the outsourcer also was affected by the powerfailure. A lot of people thought they were protected through theiroutsourcing contracts, says Vanston. Networks were down, powersources were out. You couldnt even get to the secondaryproviders.

Diesel Generators
Diesel generators have become a popular disaster recovery tool forinsurers. The majority of the major players will have a dieselbackup of some facility that allows them to run their majorcritical systems, says Barker. They may have 12 hours of hotstandby through battery storage capability, and they may haveenough fuel on site to last them for a week to run the diesels andgenerate the power.

Not all diesel generators passed the test in the blackout. Theproblem was a lot of them werent maintained, so they wouldnt start,says Vanston. Companies projected the amount of power they need,and they bought appropriate diesel generators and put in thecorrect supply limitations. The problem was they didnt do any loadtesting. A lot of companies started a big generator, ran it throughthe wires, and it just blew a bunch of circuits because it wasntseparated correctly. There are a whole bunch of minor things peopledidnt test until the actual disaster.
The generators worked fine for Virginia Farm Bureau. The buildingis wired so all the PCs can run off a generator, says Fehn. We havespecially marked plugs.

Some less-fortunate businesses learned several lessons from theblackout. Specifically, computers need electrical power. A lot ofpeople didnt know what to do because their processes or callingtrees were stored on a central server, and they couldnt get theinformation, says Vanston.

Go to the Tapes
Both GHS and Virginia Farm Bureau make backup tapes that are storedat a secure facility away from the home office. We have tapebackups stored on site and off site in fireproof cabinets, saysSkeffington.

Fehn says Virginia Farm Bureau has a warehouse facility in anotherpart of Richmond where it can operate in the event of an emergency.Weve configured it as a hot site, he says. If [the home office] waseliminated or rendered inhabitable, our contract with our networkprovider allows us to transfer all the data lines to a couple ofT1s that have been pulled to that site. We have a contract wherewell have some servers dropped in, take backup tapes and load themonto those systems, and kind of hook everything together.Hopefully, within a short period of time, were back up andrunning.

Happily for Virginia Farm Bureau, Isabel did not damage the homeoffice or the warehouse site, but the storm damage was sowidespread in the community the company is considering moving thebackup site further inland to Roanoke, in the western part ofVirginia.

If we have a fire [at the home office], its not going to be anissue for our warehouse site, says Fehn. A tornado? Probably thesame thing. A hurricane? Its big enough. [Isabel] wasnt a nice,straight, one-mile line. It pretty well knocked over everything inits path, and its path was huge. The western part of the statedidnt take the brunt of the wind, though.

Synchronize Your Data?
One result of the blackout was it made some companies believe theywere wasting money on having synchronized data stored by anoutsourcer. That really teed off a lot of people because they spenta ton of money having these protection contracts that couldnt besatisfied, says Vanston. They pay a bunch of money, and when theycant get restored, they realize that money could have been used forother things.

Companies will be looking at different scenarios in the wake of theblackout, Vanston asserts, including doing things in house, such asbuilding a secondary data center that can be used multiple ways.They can do app to app, testing, quality-assurance-type approacheson the secondary site and have it there in case of a disaster, hesays. Its been kind of a fundamental shift instead of automaticallyoutsourcing because its easier. I think people are looking at itfrom a view of if they had two architectures, could they dodifferent things?

One problem is, due to fiber channel limitations, synchronous datareplication cant be done beyond a 50- to 60-mile area, according toVanston. So now you have this widespread disaster, and peopleconsider maybe they dont need synchronous data replication; maybethey should start building architecture and applications from ana-synchronous-type approach. Then you could have geographicallyseparated data centers.

There likely would be data-integrity and quality-of-service issuesif information is sent by packets to another data center. But thereality is maybe our protection level changes, he says.

Money Drives Data
Whatever insurers do to develop a disaster recovery plan is goingto be expensive. The investment for physical plant backups, such asdiesel generators, is not a trivial number, says Barker. Vanstonagrees but sees cost concerns being tempered somewhat. Cost driveseverything, he says. But at the same time I would argue a few yearsago 90 percent of the decision was based on cost. I think thatsprobably down to 70 percent now.

The insurance industry has an advantage over other financialservices businesses because insurers dont view their data in azero-latency environments, unlike securities and bankingindustries, where huge dollar amounts flow across the system in anygiven period of time.

If [securities or banking businesses] are down for 30 seconds or aminute, that can be an extremely expensive experience, says Barker.In the insurance industry, while a lot of carriers will tell youtheir desire is to get to zero latency, the truth of the matter isthey probably are fine with one day.

That is because many insurance systems are monthly driven. Theobjective is to drive [the date] from a monthly cycle of refresh inthe warehouse down to two weeks, or one week, or in some cases downto a day, says Barker. It depends on their focus around customerservice and how closely they track transactions against a client. Iwould say the majority of insurers are in that one week to onemonth cycle range to refresh data in a warehouse.

Barker doesnt mean to imply insurers can afford to go a monthbefore knowing a policy has been paid or a claim has been accepted.But is it critical to the business if the data is not available fora period of a day or two or three? he asks.

Man-made Disasters
Another kind of disaster insurers are preparing for is the attackon their data warehouse or their systems by a hacker or through avirus. An electronic hit really can throw an organization into aspasm, says Barker. It can bring you down for days, evenweeks.

Risk managers might monitor power usage and weather reports quiteoften, but a carriers systems have to be monitored every second ofthe day. If a virus gets through your firewall, it can spreadthrough your system in a heartbeat, says Barker. Once you getinside a secure network, the controls are off. Once you get insidethe firewall, [the system] trusts you.

When an attack occurs, the insurer has to react quickly to theemergency. The different monitoring systems usually page key ITstaff, says Skeffington. If something actually gets through [thefirewall or virus scanners], a fast, planned, and careful responsehas to be ready ahead of time. You cant just go running around withyour hair on fire trying to punch it out. Thats just not going tobe prudent. Thinking it through ahead of time is key, and thenhaving the staff in place. Even though it might be a largeexpenditure, redundancy is a huge saver if something does go southon you.

Time to Recover
Some disasters are easier to recover from than others, Barker says.Sometimes its as simple as quarantining the virus on the varioussystems, he explains. If its a damaging virus, it can mean eitherformat the drive and start again fresh with a clean drive or, insome cases, you actually have to replace the PC because itsrendered inoperable.

Insurance carriers are better prepared than some industries, hebelieves. There isnt an insurance company out there that doesnthave somebody whos responsible for the security of the data, saysBarker. Then there are others that have extensive departments thatdo nothing but watch e-mail traffic, content, viruses, and so on.They manage the networks carefully.

Have a Policy
Fehn knows its a clich, but he says he learned firsthand fromIsabel: Have a plan. He never had seen a hurricane travel so farinland before Isabel. Now he is a believer. Its one of those thingsyou think cant happen to you, and sure enough it can, he says. Planthe best you can, and go through the process of regularly reviewingand fine-tuning it.

Is there such a thing as going too far in efforts to protect dataand ensure its availability to users? We havent gone overkill yet,says Skeffington. The only time people decide its not overkill iswhen they get hacked and they find out they needed all those[protective] steps, he adds.

The only real overkill is to unplug the wire and manually grantaccess to the data, he continues. That, in my opinion, would beoverkill. But any automatic system that you can put in place thatmonitors, that blocks, that scans for virusesI dont think anyapplication that can do those jobs is overkill.

Vanston says IT has the charge of carrying out the plan, but thepolicy for disaster recovery and business continuity has to bedeveloped by the business side. You want to do a business impactanalysis and understand what doesnt need to be protected, he says.Its not just a case of lets protect everything. Its a question ofwhere you really want to put the good dollars. Its a fairlymonotonous and grueling process. Thats why you need high-levelbusiness people to sign off on it and say, We cant protecteverything, so lets protect these three or four things.

Data Mining, Data Warehouse Tech Guide

ADP Claims_Services Group
San Ramon, Calif.
800-237-4968
www.adpclaims.com

Axiom Software Laboratories
New York, N.Y.
212-248-4188
www.axiomsl.com

Cargofound
Auckland, New Zealand
+64 9 817 4421
www.cargofound.com

CSC Financial Services
Austin, Tex.
800-345-7672
www.csc-fs.com

Data Advantage Group
San Francisco, Calif.
415-469-4300
www.dataag.com

DataQuick
San Diego, Calif.
888-604-3282
www.dataquick.com

DecisionPoint Applications
San Mateo, Calif.
800-970-5372
www.dpapplications.com

Delphi Technology
Cambridge, Mass.
617-494-8361
www.delphi-tech.com

Depository Trust & Clearing Corp.
New York, N.Y.
212-855-5651
www.dtcc.com

E-Z Data
Pasadena, Calif.
800-777-9188
www.ez-data.com

Fair Isaac
San Rafael, Calif.
415-472-2211
www.fairisaac.com

Fiserv
Pittsburgh, Pa.
412-577-3000
www.fiserv.com

Focus Solutions Inc.
Fort Washington, Pa.
215-643-9300
www.focus-inc.com

IBM
White Plains, N.Y.
800-426-4968
www.ibm.com

Insight Decision Solutions, Inc.
Markham, Ont., Canada
905-475-3282
www.insightdecision.com

Insurance Information Exchange
College Station, Tex.
800-683-8553
www.iix.com

Magnify
Chicago, Ill.
312-384-7161
www.magnify.com

MicroStrategy
McLean, Va.
703-848-8600
www.microstrategy.com

Millbrook, Inc.
Bethlehem, Pa.
610-867-7400
www.millbrookinc.com

Miningham & Oellerich, Inc.
New York, N.Y.
212-349-4410
www.serveme.com

MIS AG
Newark, N.J.
973-679-0724
www.mis-ag.com

NCR Teradata
Dayton, Ohio
937-445-5000
www.teradata.com

OUR-HR, LLC
Greenville, Miss.
663-334-6919
www.our-hr.com

Pinpoint Solutions, LLC
Livingston, N.J.
973-716-0700
www.pinpnt.com

Priority Data Systems
Omaha, Neb.
800-228-9410
www.priority-data.com

ReClaim Technologies
Newark, Ohio
740-344-6956
www.reclaimtech.com

Risk Laboratories
Marietta, Ga.
678-784-4600
www.risklabs.com

Sagent Technology
Mountain View, Calif.
650-815-3100
www.sagent.com

Thazar Solutions
Frisco, Tex.
972-377-1110
www.thazar.com

Universal Conversion Technologies
Addison, Tex.
214-348-2000
www.uctcorp.com