Through the American Recovery and Reinvestment Act of 2009, the federal government is investing up to $27 billion in health information technology (HIT).¹ One of the rationales for this investment is the expectation that adoption and meaningful use of HIT will reduce healthcare costs.² However, a report by the Congressional Budget Office in 2008 highlighted substantial uncertainty about the actual financial effect of HIT, saying that healthcare costs could decrease, stay the same, or increase.³ Costs could decrease if HIT reduces unnecessary utilization and reduces expensive adverse events. Costs could stay the same if HIT changes care but not in ways that introduce efficiencies. Costs could increase if HIT actually slows down providers, decreasing efficiency; leads to a more expensive, computersavvy workforce; or leads to higher utilization of medical services.³

Previous work in this area has largely modeled the financial effects of whole HIT applications, assuming that the effects of those applications were similar across different contexts.^4,5 However, this assumption may not be true, because HIT is an inherently heterogeneous intervention. Electronic health records (EHRs) and health information exchange (HIE), 2 dominant forms of HIT, are themselves applications composed of numerous functionalities that are variably implemented, configured, and/or used. This heterogeneity exists despite federal efforts to standardize the functionalities of “certified” EHRs.⁶

We sought to develop a framework that would describe more precisely the specific functionalities enabled by EHRs and HIE that may be expected to mediate any financial effects. We also sought to rank the relative importance of these functionalities for their expected financial effects, with input from national experts. Developing such a framework would have 3 main applications. First, the rankings could inform the selection of measures for Stages 2 and 3 of the federal EHR Incentive Program to promote “meaningful use.”¹ Second, the rankings could inform implementation efforts, as clinicians and hospitals choose among a “menu” of meaningful use measures.¹ Third, the rankings could inform evaluation efforts, as investigators seek to measure the actual financial impact of EHRs and HIE.

METHODS

Overview

Our methods consisted of 8 steps: 1) choosing technologies and healthcare settings, 2) identifying functionalities enabled by EHRs and HIE, 3) conducting internal ratings, 4) presenting a preliminary framework to a national expert panel, 5) modifying the ratings, 6) identifying top-scoring functionalities, 7) comparing these with the final Stage 1 meaningful use criteria, and 8) final validation.

Choosing Technologies and Healthcare Settings

We considered 2 types of health information technology: EHRs and HIE. For the purposes of this study, we considered EHRs and HIE to be distinct. We considered EHRs to be non-interoperable, that is, not including data from external sources. We considered HIE to be the electronic delivery of data from external sources, whether that delivery is through a freestanding portal or delivery into an EHR. We considered 3 healthcare settings: ambulatory, inpatient, and emergency department (ED) care. We then developed 6 technology-setting combinations (2 technologies  3 settings).

Identifying Functionalities Enabled by EHRs and HIE

We conducted a literature search to identify functionalities contained in EHRs and HIE applications. An example of such a functionality is the availability of alerts for drugdrug interactions in the context of electronic prescribing. We specifically sought functionalities that would be used by clinicians for the medical decision making that would drive healthcare costs. We included lists of functionalities generated by the Commission for the Certification of Health Information Technology (CCHIT)⁶ and the Institute of Medicine.⁷ We supplemented the literature review with functionalities encountered in the authors’ clinical, informatics, and research experiences. We populated each technology-setting combination with all relevant functionalities (eAppendix available at www.ajmc.com).

Conducting Internal Ratings

We developed a set of 3 domains upon which the functionalities would be rated: 1) probability of achieving a benefit, or the probability that the functionality will result in the desired effect in the real world for a given patient; 2) time to achieve a benefit, or the time from the “go live” date to the occurrence of the desired effect; and 3) probability of measuring a benefit, or the probability of being able to capture through research a statistically significant effect size, given available data and resources. Each domain was matched with a 3-point Likert scale, where the most desirable value had a value of 3 points. Four of the authors developed an initial set of ratings for each functionality in each technology-setting combination. The scores reflected what the raters estimated could be implemented in the next few years, rather than an assessment of what is currently implemented.

For each functionality, we created a simple sum across domains. We then selected the top 10 functionalities in each technology-setting combination, allowing more than 10 if there were ties.

Presenting a Preliminary Framework to a National Expert Panel

We convened a panel of 28 national experts from the fields of health information technology, health information exchange, health services research, healthcare economics, and healthcare policy (see Acknowledgments section). We held an in-person meeting in New York City in April 2007. The panel approved the methodology that had been used to date, added a small number of additional functionalities, and suggested the 3 additional domains: 4) complexity of implementation, or how difficult it is to “turn on the switch;” 5) likelihood of usage, or the probability that providers will actually use the functionality; and 6) expected magnitude of the financial impact, or the expected magnitude of cost savings from the payer perspective. The payer perspective was chosen, because this most closely aligns with healthcare expenditures, like those that the federal EHR Incentive Program is designed in part to address.

Modifying the Ratings