Houston experience shows value of planning for disasters
Hospital IT executives would be wise to prepare for the kind of flooding that has devastated Houston this week, particularly carefully considering the location of their servers, according to Andrew Gettinger, MD, chief medical information officer in the Office of the National Coordinator for Health IT.
Gettinger points out that floods resulting from Hurricane Sandy in 2012 were a stark reminder that the location of organizations’ data centers can lead to catastrophic consequences.
“We learned after the hurricane that went up the East Coast, devastating New York and New Jersey, that hospitals which had their data centers in the basement had outages,” he says.
That was a wake-up call for healthcare organizations that reconsidered having these highly sensitive data centers in places that were vulnerable to flooding, according to Gettinger, who notes that many hospitals “moved them up” to higher floors of a building to protect them from water damage.
Gettinger says he’s not surprised by the resiliency of hospital electronic health records systems in Houston; those EHRs have continued to operate despite widespread flooding.
“A lot depends on the thinking that has gone into setting up the EHRs,” he observes, adding that hard lessons were learned by these organizations following Hurricane Ike in 2008—as well as Tropical Storm Allison in 2001—which resulted in crippling, long-lasting effects in Houston.
It’s not enough to have a data center located on an upper floor of a building, asserts Gettinger. Backup systems also must be out of harm’s way from rising waters, preferably not in the same general location as an organization’s data center.
“If a hospital backs up all of its information to a data center down the block, which is also flooded, that’s not a sufficient solution,” adds Gettinger. “You have to think about the geography that’s likely to be at risk and make sure that your backup solution takes care of that so you can recover.”
He recommends that organizations develop broad contingency planning that takes into account elements outside of the data center, such as backup systems and backup generators. As an example of the former, Gettinger cites the case of a deadly tornado in 2011 that struck Joplin, Mo., killing 134 people, including five patients at St. John’s Regional Medical Center, which was heavily damaged.
The EHR at St. John’s “was back online the next day because they had a hospital system where all of the data was backed up to other hospitals,” notes Gettinger. “So when one hospital suffered that catastrophic loss due to the tornado, the data from that hospital was off site and available subsequently.”
EHR systems can be “real assets during natural disasters,” he adds. However, they are useless amid widespread utility outages if they are not able to run on backup generator power.
“You need electricity, and if it goes out and your backup generators don’t work, that becomes a problem,” comments Gettinger. “The magnitude of Hurricane and Tropical Storm Harvey is a great opportunity to point out that whatever your backup contingency system is, it needs to be sufficiently robust so that a disaster affecting an entire region is not going to be impacted by that.”
Cloud-based EHRs through hosted solutions offer the peace of mind of knowing that a hospital’s data is backed up in data centers in other parts of the country—far from a natural disaster. “It’s then very easy to get access to the data,” says Gettinger. “So, it’s an argument for not only the backup but operationally functioning that way.”
For its part, EHR vendor Epic maintains an on-campus production data center in Verona, Wis., which the company claims is impervious to tornadoes—a potential risk in the Midwest. In addition, the data center’s backup generators can power the facility in the case of a blackout for several weeks.
Epic, which offers both on-premise and hosted data solutions for its customers, also purchased a Mayo Clinic data center in Rochester, Minn., used specifically as backup infrastructure for disaster recovery.
“Each organization needs to look at what they are at risk for in terms of natural disasters and develop sufficient emergency contingency plans,” concludes Gettinger. “What’s important is thinking it through in advance and having plans to mitigate what are the most likely consequences.”