Value of ED patient experience data falling short
Patient experience data is increasingly important in healthcare when it comes to public reporting and value-based payment models for inpatient hospital care. However, the data is not particularly reliable or valid, say researchers who are calling for a better process to measure, capture and report it.
Researchers at the George Washington University and US Acute Care Solutions, a staffing company, analyzed commercially generated patient experience survey data from 2012 to 2015—collected from a large sample of emergency departments—and found that both physician- and facility-level patient experience data varied greatly month to month.
In particular, their analysis of a total of 1,758 facility-months and 10,328 physician-months of data revealed that physician variability is considerably higher than facility variability.
“As a physician, if there’s something about the way I’m practicing medicine that produces a certain level satisfaction of the patients that assess me, then from month to month, my scores should be relatively stable,” says Jesse Pines, MD, director of the Center for Healthcare Innovation and Policy Research and professor of emergency medicine at the GW School of Medicine and Health Sciences.
However, that’s not what Pines and his colleagues found when they evaluated satisfaction surveys gathered from patients about their experience in the ED with questions on how they perceived their physician and the facility. “What we actually found was that the scores bounced around tremendously,” he adds.
Across facility data, 40.8 percent had greater than 10 points of percentile change, 14.7 percent changed greater than 20 points, and 4.4 percent changed greater than 30. Across physician data, 31.9 percent changed greater than 20 points, 21.5 percent changed greater than 30, and 13.6 percent changed greater than 40.
Pines contends that the results of the study, published in the Annals of Emergency Medicine, has important implications because this kind of patient experience data is used to judge physician performance and hospital performance, as well as drives managerial decisions such as compensation and employment, and how a hospital is perceived through public reporting.
“The concern is that there’s a problem with the way that this information is being measured, and specifically that it’s just not a reliable way of gathering information about what sort of patient experience that a particular physician delivers,” he says.
In addition, Pines notes that if the patient experience data currently being collected is not particularly reliable or valid, then it will negatively impact the industry’s implementation of the Medicare Access and CHIP Reauthorization Act. Under MACRA, participating providers will be paid based on the quality and effectiveness of the care they provide.
While the study analyzed commercially-generated patient experience survey data from Press Ganey, Pines says it’s critical to evaluate how federal government programs are utilizing this kind of information and if the same data problems exist in pay-for-performance and other public reporting.
“The goal of quality measurement is to really get a good assessment of how we’re delivering service,” he observes. However, Pines points out that one of the problems with the patient experience data in the study was the low response rate (3 percent to 16 percent) in patient surveys.
“Imagine you conduct a survey, and only the very happy and very unhappy return their surveys,” said senior author Arvind Venkat, MD, chair of research at US Acute Care Solutions. “What you get is a very biased sample. That makes it difficult to come to any meaningful conclusions from the data.”
To increase patient response rates, Pines says surveys need to be shorter. “Currently, the survey is more than 70 questions and can take a considerable amount of time.”
Other ways to boost patient participation is to deliver surveys via smartphone and other mobile devices to make it easier to respond, Pine concludes. “Also, if we can get patients to fill out the surveys before they leave or as they’re leaving the emergency department, they may actually better remember their experience and be able to provide more detailed feedback.”
However, Press Ganey has significant concerns about this study, arguing that the variability is a function of the volume of data used in the analyses for monthly estimates and monthly provider estimates of performance.
“Roughly half of all patients enter the hospital through the emergency department. It’s a critical setting to collect deep patient experience and evaluate care,” said Deirdre Mylod, senior vice president of analytics and solutions for Press Ganey, as well as executive director of the Institute for Innovation. “We understand the challenges associated with capturing patient experience data in the ED. However, without adequate sample sizes, variability like that observed by the researchers is likely to occur. For this reason, both CMS and Press Ganey have minimum standards for numbers of surveys, which were not considered in this research. This study demonstrates why capturing the voice of every patient is critical in order to gain a true picture of the patient experience.”