Which Components of Health Information Technology Will Drive Financial Value?
Lisa M. Kern, MD, MPH; Adam Wilcox, PhD; Jason Shapiro, MD; Rina V. Dhopeshwarkar, MPH; and Rainu Kaushal, MD, MPH
Through the American Recovery and Reinvestment Act of 2009, the federal government is investing up to $27 billion in health information technology (HIT).1 One of the rationales for this investment is the expectation that adoption and meaningful use of HIT will reduce healthcare costs.2 However, a report by the Congressional Budget Office in 2008 highlighted substantial uncertainty about the actual financial effect of HIT, saying that healthcare costs could decrease, stay the same, or increase.3 Costs could decrease if HIT reduces unnecessary utilization and reduces expensive adverse events. Costs could stay the same if HIT changes care but not in ways that introduce efficiencies. Costs could increase if HIT actually slows down providers, decreasing efficiency; leads to a more expensive, computersavvy workforce; or leads to higher utilization of medical services.3
Previous work in this area has largely modeled the financial effects of whole HIT applications, assuming that the effects of those applications were similar across different contexts.4,5 However, this assumption may not be true, because HIT is an inherently heterogeneous intervention. Electronic health records (EHRs) and health information exchange (HIE), 2 dominant forms of HIT, are themselves applications composed of numerous functionalities that are variably implemented, configured, and/or used. This heterogeneity exists despite federal efforts to standardize the functionalities of “certified” EHRs.6
We sought to develop a framework that would describe more precisely the specific functionalities enabled by EHRs and HIE that may be expected to mediate any financial effects. We also sought to rank the relative importance of these functionalities for their expected financial effects, with input from national experts. Developing such a framework would have 3 main applications. First, the rankings could inform the selection of measures for Stages 2 and 3 of the federal EHR Incentive Program to promote “meaningful use.”1 Second, the rankings could inform implementation efforts, as clinicians and hospitals choose among a “menu” of meaningful use measures.1 Third, the rankings could inform evaluation efforts, as investigators seek to measure the actual financial impact of EHRs and HIE.
Our methods consisted of 8 steps: 1) choosing technologies and healthcare settings, 2) identifying functionalities enabled by EHRs and HIE, 3) conducting internal ratings, 4) presenting a preliminary framework to a national expert panel, 5) modifying the ratings, 6) identifying top-scoring functionalities, 7) comparing these with the final Stage 1 meaningful use criteria, and 8) final validation.
Choosing Technologies and Healthcare Settings
We considered 2 types of health information technology: EHRs and HIE. For the purposes of this study, we considered EHRs and HIE to be distinct. We considered EHRs to be non-interoperable, that is, not including data from external sources. We considered HIE to be the electronic delivery of data from external sources, whether that delivery is through a freestanding portal or delivery into an EHR. We considered 3 healthcare settings: ambulatory, inpatient, and emergency department (ED) care. We then developed 6 technology-setting combinations (2 technologies î€³ 3 settings).
Identifying Functionalities Enabled by EHRs and HIE
We conducted a literature search to identify functionalities contained in EHRs and HIE applications. An example of such a functionality is the availability of alerts for drugdrug interactions in the context of electronic prescribing. We specifically sought functionalities that would be used by clinicians for the medical decision making that would drive healthcare costs. We included lists of functionalities generated by the Commission for the Certification of Health Information Technology (CCHIT)6 and the Institute of Medicine.7 We supplemented the literature review with functionalities encountered in the authors’ clinical, informatics, and research experiences. We populated each technology-setting combination with all relevant functionalities (eAppendix available at www.ajmc.com).
Conducting Internal Ratings
We developed a set of 3 domains upon which the functionalities would be rated: 1) probability of achieving a benefit, or the probability that the functionality will result in the desired effect in the real world for a given patient; 2) time to achieve a benefit, or the time from the “go live” date to the occurrence of the desired effect; and 3) probability of measuring a benefit, or the probability of being able to capture through research a statistically significant effect size, given available data and resources. Each domain was matched with a 3-point Likert scale, where the most desirable value had a value of 3 points. Four of the authors developed an initial set of ratings for each functionality in each technology-setting combination. The scores reflected what the raters estimated could be implemented in the next few years, rather than an assessment of what is currently implemented.
For each functionality, we created a simple sum across domains. We then selected the top 10 functionalities in each technology-setting combination, allowing more than 10 if there were ties.
Presenting a Preliminary Framework to a National Expert Panel
We convened a panel of 28 national experts from the fields of health information technology, health information exchange, health services research, healthcare economics, and healthcare policy (see Acknowledgments section). We held an in-person meeting in New York City in April 2007. The panel approved the methodology that had been used to date, added a small number of additional functionalities, and suggested the 3 additional domains: 4) complexity of implementation, or how difficult it is to “turn on the switch;” 5) likelihood of usage, or the probability that providers will actually use the functionality; and 6) expected magnitude of the financial impact, or the expected magnitude of cost savings from the payer perspective. The payer perspective was chosen, because this most closely aligns with healthcare expenditures, like those that the federal EHR Incentive Program is designed in part to address.
Modifying the Ratings
We added internal ratings for the additional functionalities and additional domains. We also had 7 national experts review the ratings in detail and suggest possible edits. Suggestions were reviewed and reconciled through consensus by 3 authors.
Identifying Top-Scoring Functionalities
For each functionality, we summed scores across the 6 domains, for a possible score of 6 to 18 points. We determined the distribution of scores for each technology-setting combination. None of the distributions were clearly bimodal with obvious cut-points for the highest scoring functionalities. Thus, we selected and applied the cutoff that would yield approximately 10 high-scoring functionalities for each technology-setting combination. We also used 2-tailed t tests to compare the average score for EHR functionalities with the average score for HIE functionalities. We used analysis of variance (ANOVA) to compare the average scores across healthcare settings.
Comparing With the Final Stage 1 Meaningful Use Criteria
We compared the top-scoring functionalities with the final Stage 1 meaningful use criteria. Stage 1 includes 15 “Core” measures that are required for all eligible providers and hospitals, such as “record patient demographics,” “record vital signs and chart changes,” and “use computerized order entry for medication orders.”1 In addition, Stage 1 includes 12 “Menu” measures, from which eligible providers and hospitals are expected to choose 5.1 Examples of Menu measures include “implement drug formulary checks” and “perform medication reconciliation between healthcare settings.” We calculated the percentage of top-scoring functionalities from our framework that are part of meaningful use and analyzed the content of those that are not yet part of meaningful use, in order to identify opportunities for future meaningful use measures.
We presented the top-scoring functionalities to experts again in August 2011. They validated the final set and recommended no changes, as they believed that it was consistent with and went beyond Stage 1 meaningful use.
We identified 105 unique functionalities enabled by EHRs and HIE and 233 functionality-setting combinations (Appendix). We identified a total of 84 functionalities for ambulatory care, 80 for inpatient care, and 69 for ED care. We identified a total of 160 functionality-setting combinations for EHRs and 73 for HIE. Overall and within each setting, there were more functionalities for EHRs than HIE.
Overall, the average summary score for each functionality in each setting was 12.5 (median 13, standard deviation [SD] 2.6) on a scale from 6 to 18, in which higher scores represented a higher likelihood of having a measurable positive financial effect. The average functionality for EHRs scored significantly higher than the average functionality for HIE (13.0 vs 11.3, P <.0001). There were no differences in average scores across healthcare settings (P = .33).
The distribution of scores is shown in Figure 1 for EHRs and Figure 2 for HIE. The cut-point that yielded approximately 10 high-scoring functionalities per technology-setting combination was a score of >16 for EHRs and >13 for HIE. Using this threshold, there were a total of 31 unique high-scoring functionalities and a total of 54 high-scoring functionality-setting combinations (Table). For EHRs, the high-scoring functionalities had scores ranging from 16 to 18, with a mean of 16.5 (SD 0.7). For HIE, the high-scoring functionalities had scores ranging from 13 to 16, with a mean of 13.9 (SD 1.0).
For EHRs in particular, there were 15 unique high-scoring functionalities and 27 high-scoring functionality-setting combinations (Table). Examples of high-scoring EHR functionalities included: providing alerts for expensive medications (ambulatory and inpatient care), providing alerts for redundant lab orders (inpatient and ED care), and displaying imaging results (ED). For HIE, there were 16 unique high-scoring functionalities and 27 high-scoring functionality-setting combinations (Table). Examples of high-scoring HIE functionalities included: sending and receiving imaging reports (ambulatory, inpatient, and emergency care), receiving laboratory results (ambulatory and emergency care), and enabling structured medication reconciliation.
All of the Stage 1 meaningful use measures reflect functionalities that were scored in our framework. Of the 15 Core meaningful use measures, 4 were ranked highly in our framework as having the most potential for driving financial value: use computer provider order entry (CPOE) for medication orders, implement drug-drug interaction checks, implement the capability to electronically exchange key clinical information among providers and patient-authorized entities, and report clinical quality measures to the Centers for Medicare & Medicaid Services or the states. Of the 12 Menu meaningful use measures, 4 were ranked highly in our framework: implement drug formulary checks, incorporate clinical laboratory test results into EHRs as structured data, perform medication reconciliation between care settings, and provide summary of care record for patients referred or transitioned to another provider or setting.
Of the 54 high-scoring functionality-setting combinations in our framework, 25 (46%) are represented in Stage 1 meaningful use (Table). Thus, nearly half of the functionality- setting combinations in our framework align with Stage 1 meaningful use and represent the portion of Stage 1 meaningful use that is most likely to yield financial benefits. Of the functionality-setting combinations that were not represented in Stage 1, some may be implemented as prerequisites to the formal definition of Meaningful Use but are not stated as measures per se, such as default drug dosages and alerts for preventive services. Many others are distinct and represent measures directed squarely at utilization and costs rather than at healthcare quality: provide alerts regarding generic substitution, provide rules-driven financial and administrative coding assistance, provide alerts for expensive medications, provide alerts for laboratory charges, and provide alerts for redundant lab orders.
We developed a framework for rating the potential financial effects of functionalities enabled by EHRs and HIE. We identified 105 unique functionalities applicable across 3 healthcare settings. We found more functionalities for EHRs than for HIE. This finding is consistent with CCHIT certification criteria6 and likely due to the medical community’s greater familiarity with EHRs than HIE. We found that EHR functionalities were rated more highly than HIE functionalities. This was driven, in part, by the experts’ views that implementing HIE was more complex and would take longer than implementing EHRs. Thus, the financial effect of HIE might be seen on a longer time horizon than that of EHRs alone.
In addition, we found that top-scoring functionalities for EHRs were generally more sophisticated than top-scoring functionalities for HIE. Top-scoring functionalities for HIE relied on the exchange of clinical data among providers.8 Top-scoring functionalities for EHRs frequently went beyond the possession of data to include decision support, or computer-generated information at the point of care.9 What is still emerging—and thus not yet reflected here—is decision support in the context of HIE. An example of this would be alerts for primary care physicians regarding receipt of expected consultation feedback from specialists to whom they referred their patients, along with the ability to send electronic reminders to those specialists. It is very possible that the financial return on EHRs and HIE may be greatest in this intersection of decision support and care coordination.
This study did not assume any particular architecture for HIE, such as a free-standing Internet-based portal or direct feeds into EHRs.10 This study indicates which functionalities are most likely to have a measurable positive financial effect, but the ratings do not yet reflect the differential costs of implementing different HIE architectures, which are not yet known.
Previous work in this area has included high-level models for estimating the likely financial impact of HIT.4,5 These models, which were based on expert opinion, were global, estimating the impact of the whole technology, without respect to which aspects of that technology might be more or less important for achieving the desired outcome. Other studies in this area have similarly considered whole technologies or considered 1 component at a time.11-13 Our work adds to the literature by assessing the relative potential effects of different functionalities within technologies, thereby creating a framework for prioritizing different functionalities for their potential financial effects.
Finding the financial effects in HIT may be more difficult than finding the financial effects of IT in other industries. This is, in part, because the provision of healthcare is highly complex and involves “knowledge workers” who have a high degree of autonomy and provide judgment and individual expertise.14 IT does not necessarily increase productivity among knowledge workers, unless the technology is specifically tailored to their particular information needs.14 Other industries with more routinized, transaction-based businesses may be made more efficient more easily by computer systems that automate decisions. This distinction from other industries underscores the importance of having a framework for evaluating the impact of IT on healthcare.
This study has several limitations. First, the results are based on expert opinion, rather than empirical observations; they are meant to inform future quantitative evaluation. Second, the functionality approach is somewhat artificial in that functionalities are not generally implemented 1 at a time, but rather as a package. It is still important to look at the effects of HIT interventions as a whole, because combinations of functionalities may yield positive or negative effects beyond those of individual functionalities. Nevertheless, the next generation of studies evaluating the effects of HIT will need to look at usage of particular functionalities and link them to specific outcomes, as the “black box” approach to evaluating an entire electronic application at once has yielded mixed or even negative results, despite strong hypotheses of a positive effect.15-18 Third, the experts rated each functionality-setting combination on its own potential merit; they were not asked to explicitly compare or rank functionalities against each other. It is possible that an explicit ranking task would have yielded somewhat different results and would have addressed the issue of needing a somewhat arbitrary cutoff for high-scoring functionalities. However, the final set of functionalities has face validity, including consistently high ratings of some functionalities across multiple healthcare settings. Finally, this framework considers financial effects from the payer perspective; separate ratings could be derived for the provider, patient, and societal perspective, and those could have different results.
This study has important policy implications. First, this study suggests specific measures that could be incorporated into Stages 2 and 3 of meaningful use. In particular, this study suggests that Stages 2 and 3 should include measures more explicitly focused on utilization and costs, such as: provide alerts regarding generic substitution, provide rules-driven financial and administrative coding assistance, provide alerts for expensive medications, provide alerts for laboratory charges, and provide alerts for redundant lab orders. Second, this study can also help clinicians and hospitals choose among a “menu” set of measures, potentially preferentially selecting those menu set measures that also received high scores in this framework. Specifically, this study suggests that eligible providers and hospitals prioritize the following 4 measures when choosing among those in the Menu Set: implement drug formulary checks, incorporate clinical laboratory test results into EHRs as structured data, perform medication reconciliation between care settings, and provide summary of care record for patients referred or transitioned to another provider or setting. Third, because this study was conducted from the perspective of the payer and because high-scoring functionalities were found that suggest the potential for financial benefit from the payer perspective, there are additional implications for providers. Although this study does not address the provider’s financial perspective directly, providers are generally responsible for the costs of EHR and HIE implementation. Thus, adding high-scoring functionalities to the federal government’s incentive program could appropriately subsidize the cost of the technology for providers who might otherwise not invest in these technologies if they do not receive the financial benefit of them. Fourth, the high-scoring functionalities from this study could be used to guide any EHR or HIE implementation, independent of the meaningful use program, because these functionalities represent the areas in which experts expect the most financial and clinical value. Finally, this study can help shape quantitative evaluations of the actual financial effects of EHRs and HIE as they are implemented.
The relevance of this functionality-based study to the financial effects of whole EHR and HIE products also relates to the issues of implementation, configuration, and use. Although EHRs and HIE products are adopted as whole applications, different functionalities may be turned on or off by individual technicians and users of these systems. Thus, our finding that some functionalities have more potential for positive financial effects than others matters. If whole applications are adopted, but the most promising functionalities are turned off, then the likelihood of a positive financial effect for the whole application is low. Previous studies have shown that many clinicians have been found to adopt EHRs without activating or using decision support,19 which our study suggested would be a prime mediator of a positive financial effect.
In conclusion, EHRs are most likely to generate a positive financial effect through the use of clinical decision support. HIE is most likely to generate a positive financial effect through its ability to coordinate care among providers. Adding decision support to HIE could potentially yield even greater financial returns. Implementing Stage 1 meaningful use is likely to yield positive financial effects, but the largest positive financial effects may be still to come.