As we, as well as, people from around the country watched the storm, and now the aftermath of Hurricane Sandy on the news, our thoughts and prayer go out to those in recovery mode.
How Did Data Centers Fare During the Storm?
It was reposted that a few data centers on the East Coast were affected, and several websites went offline as a result. Huffington Post, BuzzFeed and Gawker all suffered outages. Cloud data centers and others who had enacted plans to keep their customers up and running even under the worst case scenario seemed to fare well.
Unfortunately, some of the worst hit aren’t likely to get a refuel, or consistent access to power in the immediate future, so there is something to be said for redundancy, failover and colocation. One of the worst situations right now is the flooding of 75 Broad Street in Manhattan which has taken down both Peer1 Hosting and Internap. That data center is right in the middle of the worst damage in New York, and it had to go into controlled shutdown due to lack of fuel and flood damage. And what is happening to the data?
Equinix also reported having “minor leaks” and power outages at its New York and Washington, D.C. data centers, and were at times relying on 48-hour supplies of fuel for generators. If they’re still running on emergency power, they must be coming close to running out of fuel by now.
Although cloud is a viable option for business continuity–and quite likely the best option in many cases–without proper planning, business could be disastrously affected by major disasters such as the one caused by Hurricane Sandy.
It’s certain that Hurricane Sandy is putting cloud data centers to the test. Many lessons are likely to be learned in business continuity and disaster planning once the dangers of Sandy have passed.
Advance Disaster Recovery Planning is Key
Steven Ross of Risk Masters in an article for SearchDisasterRecovery, gave a few tips on disaster recovery planning for hurricanes.
Here are a few of them:
Damage from a hurricane comes from more forces than just the big storm, including winds falling and rising water, as the storm passes.
Data center managers need to understand the applications their data center supports and use that knowledge as basis for their DR plan. This way, management can focus on the major applications in the data center and prepare their business users for short- and long-term outages.
To prepare data center staff for a hurricane season, regular drills need to be conducted to practice emergency procedures. Data center managers also have to be prepared for many of the staff not being there during a hurricane, since many employees don’t come to work when there is a hurricane warning.
Data center managers have to be prepared for destruction, damage, inaccessibility and loss of utility power. Beyond DR planning, there are also actions data center managers need to take immediately at the onset of a hurricane season.
Here they are, according to Ross:
- Board up glass walls
- Ensure generator fuel tanks are full
- Ensure heavy weather supplies, such as polyethylene sheeting, are on hand to cover equipment
- Have multiple ways to reach all staff through normal channels
- Notify equipment vendors so they prepare for emergency shipment of necessary gear
- More data center disaster preparedness tips came from IBM:
- Validate your data backup plan – Verify that your data is out of harm’s way and/or is accessible to your recovery location. Consider using a cloud service to store key data and allow your organization more flexibility to respond to changing conditions with minimal interruption to the business.
- Consider employees and the personal impact of a disaster – A company’s most important asset are their people, but the most important asset for people are their families. Consider how you would move them and their families if required, think about providing financial support to your employees during a crisis event, and consider offering counseling to help them deal with the aftermath of the crisis.
- Develop various ways to communicate with employees, partners – After people, the next most important element is communication. Communications efforts must be timely, clear and honest, as miscommunication can make a disaster even worse. Consider how you would communicate with your employees, partners, clients, media, industry, and vice versa, what training you have provided, what tools are you using and — very important – test the communications plan.
- Think about the “domino effect” when considering business risk – Years of experience monitoring regional disasters has shown that these events often create other events. For example, a hurricane normally has high winds and heavy rains that can lead to flooding, structural damage, power outage, telecommunication and/or travel disruptions.
- Plan for catastrophic events that could last a while – For example, businesses must consider the impact if the duration of the disruption to the facility, network, technology, or people is longer than a period of three days, one week, etc. Over the past decade, we have seen more devastating disaster events with a longer term duration and financial impact. Companies need to consider their options if their primary environment or key people are not available for more than two weeks.
- Think broadly – Each company is part of a supply chain or network. While you may do everything right, if you have a critical partner, supplier, vendor or provider of service, your preparedness is only as good as those other businesses.