The American Journal of Managed Care Special Issue: Health Information Technology
Leveraging EHRs to Improve Hospital Performance: The Role of Management
Recent studies fail to find a consistent relationship between adoption of electronic health records (EHRs) and improved hospital performance. We sought to examine whether the quality of hospital management modifies the association between EHR adoption and outcomes related to cost and quality.
Retrospective study of a random sample of US acute care hospitals.
Management quality was assessed via phone interviews with clinical managers predominantly from cardiac units in a random sample of 325 hospitals using a validated scale of management practices in 4 areas: operations, performance monitoring, target setting, and talent management. American Hospital Association InformationTechnology Supplement data captured whether or not these hospitals had at least a basic EHR. Acute myocardial infarction (AMI) outcomes included risk-adjusted 30-day mortality, average length-of-stay, and average payment per discharge measured using MedPAR data. Ordinary least squares regressions assessed whether management quality modifies the relationship between EHR adoption and AMI outcomes.
While we found no association between EHR adoption and our outcomes, management quality modified the relationship in the predicted direction. For length of stay, the coefficient on the in- teraction between EHR and management was –1.48 (P = .05) and for payment, it was – 7786.74 (P = .014). We did not find strong evidence of effect modification for mortality (coefficient = –0.05; P = .37).
Coupled with ongoing policy efforts to achieve nationwide EHR adoption is a growing unease that our national investment may not result in better, more efficient care. Our study is among the first to offer empirical evidence that management quality may help explain why some hospitals see substantial gains from EHR adoption while others do not.
Am J Manag Care. 2014;20(11 Spec No. 17):SP511-SP519
The US healthcare system is in the midst of an ambitious nationwide effort to adopt and use electronic health records (EHRs). Under the 2009 Health Information Technology for Economic and Clinical Health(HITECH) Act, hospitals that implement EHRs and demonstrate that they meet federally defined meaningful use criteria are eligible for financial incentive payments.1,2 HI-TECH was motivated by widespread agreement that the use of EHRs would result in substantial performance improvement. Key EHR functions—easier access to patient medical records, electronic medication ordering, and clinical decision support—should all lead to better care, less duplication, and ultimately, more efficient care. Unfortunately, the evidence to date on the impact of widespread EHR uptake has been somewhat disappointing,3-6 and suggests that the average effect of EHR adoption may be close to zero. These results have been puzzling, giving rise to the idea that factors beyond the adoption of EHRs may be influential. In industries such as retail and banking, the degree to which organizations are well managed is an important determinant of whether performance improves after adopting new forms of information technology (IT).7,8 It is possible that a similar phenomenon is at play in healthcare. The intuition that management should matter is simple: IT is a tool that influences performance based largely on how it is used, and without a concerted and thoughtful approach to managing how IT is leveraged to improve performance, its use will not achieve the potential quality and efficiency gains. In fact, implementing IT systems in poorly managed organizations may be disruptive in ways that cause productivity losses. However, the empirical evidence for this hypothesis in healthcare settings is largely lacking.
In this paper, we leverage a unique data set on hospital management practices collected through interviews with managers in a national random sample of acute care hospitals in order to explore whether management quality modifies the relationship between EHR adoption and hospital outcomes. We focus on outcomes related to acute myocardial infarction (AMI) because management survey interviewees were predominantly from cardiology units. We hypothesize that in well-run institutions, EHRs would be associated with better performance, while in poorly run institutions, EHR adoption may be associated with worse performance.
Data, Measures, and Sample Our study uses 4 sources of data. To determine whether a hospital had an EHR, we used the American Hospital Association (AHA) IT Supplement that was administered to all AHA member hospitals in 2009 (with a response rate of 69%) and captures IT functions in place at the end of 2008.9 We determined which hospitals had at least a basic EHR using the Jha et al (2009) definition of the 10 specific functions that comprise a basic EHR, such as computerized provider order entry (CPOE) for medications, test result viewing, problem lists, and medication lists.10 We used the broader AHA Annual Survey for data on key hospital characteristics: size (<100 beds, 100-399 beds, 400+ beds), teaching status (major teaching, minor teaching, nonteaching), ownership (private, for-profit; private, not-for-profit; public), census region, urban location, percent of discharges from patients with Medicare, and whether or not the hospital was part of a system.
To capture management quality, we used data from the World Management Survey that was collected in 2009 from a national random sample of 325 AHA-member acute care hospitals. Management practices were measured using a team of trained analysts who conducted phone interviews with unit or departmental managers in cardiac or orthopedics units in sample hospitals. Interviews were conducted between March and October 2009 to capture management practices in 4 broad areas, each of which could potentially influence how an EHR is used in a given hospital (eAppendix Table A1, available at www.ajmc.com). The first area, operations management, captures the extent to which primary clinical procedures are standardized and how well the clinical staff is trained in these protocols. The second area, performance monitoring, focuses on how well the organization’s performance monitoring system informs day-to-day operations of the hospital. The third area, target setting, captures how tightly targets are linked to the hospital’s wider objectives and how well targets cascade down and are clear to employees. The last area is talent management, which captures various dimensions of human capital management, including the types of systems that are in place to recruit hospital staff, whether the hospital evaluates employees and rewards them based on performance, and whether there is an active promotion system that strives to retain and incentivize employees.
The tool applies definitions and scores from 1 (“worst practice”) to 5 (“best practice”) along 20 dimensions across the 4 areas listed above (operations, performance, targets, and talent), with an average of the 20 scores serving as the overall management score. In other words, a score of 5 represents excellent management practices across all areas. In eAppendix Table A2, we provide a list of sample questions for each area as well as information for accessing the complete survey instrument. The reliability of these measures and their robustness to many different forms of psychological bias are described by Bloom and Van Reenen (2007).11 Interviewers were extensively trained in advance on hospital management, and a double-blind technique was employed in which interviewers were not told in advance about the hospital’s performance and interviewees were not told in advance how their answers would be scored. The data set also included interview and interviewee characteristics, such as the duration of the interview and the tenure of the interviewee in their position.
To capture both cost and quality performance of hospitals in our sample, we examined 3 risk-adjusted outcomes: 30-day mortality, average length of stay (LOS), and average payment per discharge (what the hospital received in payment for the stay). Risk adjustment accounted for patients’ age, sex, and comorbidities (using the Elixhauser Index). All measures were generated from Medicare data for fee-for-service beneficiaries 65 years or older who were hospitalized for AMI. We used 2009 as our focal year, but we also examined 2010 to assess whether associations persisted 1 year later. We chose to focus on AMI outcomes because we were concerned that there might be substantial unit-to-unit variation in both management and outcomes within hospitals. Since the majority of interviewees in the management survey represented cardiology units, AMI-focused outcomes were thought to be the most reliable indicators to examine the association with management quality and EHR adoption. Our analytic sample is the 191 US acute care hospitals that participated in the management survey and that responded to AHA IT Supplement in 2009. In eAppendix Table A3, we compare our analytic sample with the broader AHA sample on key hospital characteristics.
To assess whether management quality modifies the association between EHR adoption and hospital performance, we estimated ordinary least squares regressions in which we predicted (a) 30-day AMI mortality rate, (b) average LOS for AMI discharges, and (c) average payment for AMI discharges. Our focal predictors are the hospital’s management score (range of 1-5) and whether or not the hospital had adopted at least a basic EHR. We include a set of control variables to account for factors that may confound the relationship among management quality, EHR adoption, and outcomes: size, teaching status, ownership, urban versus rural location, percent Medicare discharges, and whether the hospital is affiliated with a larger system. All our models include hospital-level weights that adjust for potential nonresponse bias due to known differences between respondents and nonrespondents to the AHA IT Supplement. Weights are derived by estimating the propensity to respond to the survey using a logistic regression model with key hospital characteristics as predictors, and the inverse of this propensity value serves as the weight.10
To assess whether management quality modifies the association between EHR adoption and our outcomes, we included a term that interacts the management score with EHR adoption. We also calculated and graphed predictive margins across the range of management scores in increments of 0.5 (approximately equal to the standard deviation of 0.55 in our sample, eAppendix Table A4) in order to compare predicted outcomes for hospitals with and without EHRs.
We also conducted a set of robustness tests to assess whether our results change under different specifications. First, we included in our models a set of variables that capture potential confounders driven by differences in interview-level characteristics. Specific variables included the duration of the interview (in minutes), the reliability of the interviewee (assessed by scoring both their knowledge of management practices and their willingness to reveal infor- mation), whether the interviewee was a nurse or a physician, the seniority of the interviewee, and the number of years that the interviewee had been in their current position. Our second set of robustness tests relaxed the requirement that a hospital have all the functions that comprise a basic EHR and examined 3 specific EHR functions: test result viewing, CPOE, and decision support. Our third and final set of robustness tests examined each individual area of management independently: operations management, performance monitoring, target setting, and people management. This enables us to assess whether any observed relationships are primarily driven by 1 particular area of management.
Of the 191 hospitals in our sample 12% had adopted at least a basic EHR as of the end of 2008 (eAppendix Table A4).11 The average management score of 3.01 (SD = 0.55; range 1.4-4.3) was effectively at the midpoint of the 1-5 scale. For our primary outcomes of interest, the 30- day mortality rate was 19% (SD = 17%; range 0%-100%). The average risk-adjusted LOS for patients hospitalized for AMI was 4.89 days (SD = 1.99; range 0.86-10.38) and average payment per discharge was $12,544 (SD = $7414; range $995-$41,955).