Formal ROI Analysis Is Rare

Howard J. Anderson,
Executive Editor
Health Data Management Magazine, December 1, 2007

When El Centro (Calif.) Regional Medical Center contemplated investing in an emergency department information system, the hospital's chief nursing officer and other executives demanded a precise quantification of the potential return on investment.

"That was a first for us," says John Gaede, director of information services at the 165-bed hospital. "Now our board has demanded that we do the ROI exercise for all other systems we buy."

Indeed, it's extremely rare for hospitals and clinics to set ROI benchmarks before acquiring software and then actually measure the application's performance against those expectations, many consultants say.

"It has something to do with the culture of health care and hospitals," laments Bruce Eckert, senior manager at Beacon Partners, a Weymouth, Mass.-based consulting firm. "The not-for-profit culture is to avoid creating the business case and having the accountability that a for-profit business might."

Another consultant, however, says that situation is beginning to change.

"Hospitals know they have to invest in clinical information systems to improve the quality of care, but measuring the actual results has dragged behind the recognition that they have to put the system in," says Nancy Chapman, a partner at ACS Healthcare Solutions, a Dearborn, Mich.-based consulting firm. "Hospitals, however, are beginning to recognize that they have to demonstrate how a software investment will improve care."

With a clinical system ranging in price from $2 million for a small hospital to $15 million or more for a large facility, executives are beginning to demand quantifiable evidence of improvement in patient care, Chapman says.

"Some of our clients make a very good business case for a software investment, but then don't do as good a job checking whether the benefits were achieved," says Barbara Cox, senior principal and senior researcher at the Center for Health Information, the research unit of Noblis Inc., a Falls Church, Md.-based consulting firm. "I can't think of a client that's done the entire process very well."

Health care organizations attempting to create ROI benchmarks should form a team of clinicians, I.T. experts and others to tackle the challenging task, Cox advises.

"It takes time to put the right team together to figure out the goal of the project and what type of approaches are needed to create a baseline of measurable information," she says. "It takes a little bit of effort and some research to determine the benefits that can be achieved and determine what approach to take to measuring them."

Cox stresses that CIOs should not take the lead role in measuring ROI. "Justifying the value of an investment should be the primary responsibility of the executive sponsor in the part of the organization that will benefit from the investment," she suggests.

The committee establishing benchmarks first has to understand the goals of the organization before making a business case for the investment, Cox stresses. And the business case should change over time as the goals of the organization change, she adds.

Once software is implemented, the performance against the benchmarks should be measured regularly. "A lot of organizations don't have the energy after an implementation to go back and study whether they have indeed achieved the anticipated savings," she says.

In addition, many organizations also fail to provide an adequate budget for the post-implementation analysis, she notes.

Cox and other consultants acknowledge that setting ROI benchmarks for clinical systems is far more challenging than for financial systems. But many factors, including medication errors reduction and nurse productivity improvement, can be measured.

How Complex?

Setting up ROI expectations can range from hiring a management engineer to do "time and motion" studies of staff productivity to simply setting expectations for such process improvements as reducing chart pulls.

When he was a hospital CIO, Eckert of Beacon Partners hired an engineer to conduct a time and motion study of the medication administration process, identifying the resources required. Two months after implementing a new clinical system, the engineer did a follow-up study to demonstrate how the software made the hospital more efficient.

Hiring an engineer to conduct a sophisticated study can prove costly; Eckert's hospital spent $40,000. But if an organization is spending 20 times that figure on software and sees large potential for ROI, the investment is worthwhile, he contends.

For those not interested in hiring an engineer, Beacon Partners provides simpler ROI checklists that its clients can use to measure potential benefits. For example, hospitals can measure reductions in support cost for a new system vs. an old one; declines in power consumption; savings in space devoted to paper files; and cuts in labor costs associated with tracking down paper records.

Eckert advises provider organizations to focus on an easily measurable "global indicator," such as full-time equivalent employees per occupied bed or percentage of time nurses spend on administrative tasks.

"The amount of time that nurses spend treating patients, if they don't have a clinical information system, is typically 40%," says Chapman at ACS. "The rest of their time is spent on manual documentation. So a hospital could set a goal of using software to reduce the amount of time in documentation by 15 minutes per day per nurse."

Such an improvement could enable a hospital to free up nurses' time for treating more patients, enabling the organization to open up more beds and, ultimately, generate more revenue, she notes.

Chapman argues that the ideal way to measure ROI on clinical systems is to measure improvement in clinical outcomes, which includes reductions in medical errors.

"Hospitals don't implement clinical systems to save money," the consultant stresses. "They do it to improve outcomes and gain the ability to treat more patients more effectively with fewer medication errors."

Breaking New Ground

Like many hospitals, El Centro Regional Medical Center had never conducted a formal ROI study before acquiring software. But when it considered the purchase of an emergency department system, things changed.

The hospital's new chief nursing officer was skeptical of the value of the system and demanded proof of ROI, recalls Gaede, the director of information services.

Page 1 of 3.