Currently Viewing:
The American Journal of Managed Care Special Issue: Health Information Technology
Improving Adherence to Cardiovascular Disease Medications With Information Technology
William M. Vollmer, PhD; Ashli A. Owen-Smith, PhD; Jeffrey O. Tom, MD, MS; Reesa Laws, BS; Diane G. Ditmer, PharmD; David H. Smith, PhD; Amy C. Waterbury, MPH; Jennifer L. Schneider, MPH; Cyndee H. Yonehara, BS; Andrew Williams, PhD; Suma Vupputuri, PhD; and Cynthia S. Rand, PhD
Information Retrieval Pathways for Health Information Exchange in Multiple Care Settings
Patrick Kierkegaard, PhD; Rainu Kaushal, MD, MPH; and Joshua R. Vest, PhD, MPH
The 3 Key Themes in Health Information Technology
Julia Adler-Milstein, PhD
Leveraging EHRs to Improve Hospital Performance: The Role of Management
Julia Adler-Milstein, PhD; Kirstin Woody Scott, MPhil; and Ashish K. Jha, MD, MPH
Electronic Alerts and Clinician Turnover: The Influence of User Acceptance
Sylvia J. Hysong, PhD; Christiane Spitzmuller, PhD; Donna Espadas, BS; Dean F. Sittig, PhD; and Hardeep Singh, MD, MPH
Cost Implications of Human and Automated Follow-up in Ambulatory Care
Eta S. Berner, EdD; Jeffrey H. Burkhardt, PhD; Anantachai Panjamapirom, PhD; and Midge N. Ray, MSN, RN
Primary Care Capacity as Insurance Coverage Expands: Examining the Role of Health Information Technology
Renuka Tipirneni, MD, MSc; Ezinne G. Ndukwe, MPH; Melissa Riba, MS; HwaJung Choi, PhD; Regina Royan, MPH; Danielle Young, MPH; Marianne Udow-Phillips, MHSA; and Matthew M. Davis, MD, MAPP
Adoption of Electronic Prescribing for Controlled Substances Among Providers and Pharmacies
Meghan Hufstader Gabriel, PhD; Yi Yang, MD, PhD; Varun Vaidya, PhD; and Tricia Lee Wilkins, PharmD, PhD
Health Information Exchange and the Frequency of Repeat Medical Imaging
Joshua R. Vest, PhD, MPH; Rainu Kaushal, MD, MPH; Michael D. Silver, MS; Keith Hentel, MD, MS; and Lisa M. Kern, MD
Information Technology and Hospital Patient Safety: A Cross-Sectional Study of US Acute Care Hospitals
Ajit Appari, PhD; M. Eric Johnson, PhD; and Denise L. Anthony, PhD
Automated Detection of Retinal Disease
Lorens A. Helmchen, PhD; Harold P. Lehmann, MD, PhD; and Michael D. Abràmoff, MD, PhD
Trending Health Information Technology Adoption Among New York Nursing Homes
Erika L. Abramson, MD, MS; Alison Edwards, MS; Michael Silver, MS; Rainu Kaushal, MD, MPH; and the HITEC investigators
Electronic Health Record Availability Among Advanced Practice Registered Nurses and Physicians
Janet M. Coffman, PhD, MPP, MA; Joanne Spetz, PhD; Kevin Grumbach, MD; Margaret Fix, MPH; and Andrew B. Bindman, MD
Currently Reading
The Value of Health Information Technology: Filling the Knowledge Gap
Robert S. Rudin, PhD; Spencer S. Jones, PhD; Paul Shekelle, MD, PhD; Richard J. Hillestad, PhD; and Emmett B. Keeler, PhD
The Effects of Health Information Technology Adoption and Hospital-Physician Integration on Hospital Efficiency
Na-Eun Cho, PhD; Jongwha Chang, PhD; and Bebonchu Atems, PhD

The Value of Health Information Technology: Filling the Knowledge Gap

Robert S. Rudin, PhD; Spencer S. Jones, PhD; Paul Shekelle, MD, PhD; Richard J. Hillestad, PhD; and Emmett B. Keeler, PhD
Studies of health information technology have not kept up with the evolving needs of the healthcare system. We explain how to set them straight.
Despite rapid growth in the rate of adoption of health information technology (HIT), and in the volume of evaluation studies, the existing knowledge base for the value of HIT is not advancing at a similar rate. Most evaluation articles are limited in that they use incomplete measures of value and fail to report the important contextual and implementation characteristics that would allow for an adequate understanding of how the study results were achieved. To address these deficiencies, we present a conceptual framework for measuring HIT value and we propose a checklist of characteristics that should be considered in HIT evaluation studies. The framework consists of 3 key principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder’s perspective is used. Through examples, we show how these principles can be used to guide and improve HIT evaluation studies. The checklist includes a list of contextual and implementation characteristics that are important for interpretation of results. These improvements will make future studies more useful for policy makers and more relevant to the current needs of the healthcare system.

Am J Manag Care. 2014;20(11 Spec No. 17):eSP1-eSP8
Healthcare organizations all over the United States are adopting health information technology (HIT), yet recent studies of HIT are often unhelpful to these organizations or to those making policy decisions. Most studies fail to use adequate measures of value or to report key contextual and implementation characteristics. In this commentary, we:
  • Demonstrate the problem in a sampling of published articles.
  • Propose a conceptual framework and principles for measuring value in HIT.
  • Present a checklist of key study characteristics.

Readers can use this article as a touchstone for assessing the utility and relevance of published research studies.
In 2010, the United States Congress enacted the Health Information Technology for Economic and Clinical Health (HITECH) Act, thereby dedicating roughly $27 billion to eligible healthcare providers who demonstrate “meaningful use” of health information technology (HIT). This action motivated the purchase and installation of tens of thousands of HIT products across the US healthcare system and placed a national spotlight on HIT. The HITECH Act was justified, in part, by early studies that documented positive effects on healthcare quality, safety, and efficiency at a subset  of institutions that were early to implement HIT, and by policy analyses that extrapolated the implications of these potential benefits throughout the US healthcare system.1,2 These studies suggested that HIT had the potential to create enormous value by facilitating improvements in the quality of healthcare and reductions in costs.

However, systematic reviews of HIT have found that the evidence for value is inconclusive and that existing studies suffer from major limitations.3-6 This finding is true even of the most recent literature reviews, despite a greatly increasing number of studies evaluating HIT.6 In this paper, we suggest a way to overcome these deficiencies and make HIT evaluations more relevant to the current needs of the healthcare system, by presenting a conceptual framework for measuring the value of HIT, examining how a sample of published HIT articles report the information needed to make meaningful assessments of value, and proposing a set of criteria for future evaluations that would make them more useful for policy makers.

This paper was sponsored by the Office of the National Coordinator for Health Information Technology, which otherwise had no role in the conduct or writing of the paper or the decision to publish it.


Published studies have evaluated HIT using many different measures, including cost savings,7 time savings,8,9 utilization reduction,10-12 process adherence improvements,13,14 health outcomes,15,16 safety improvements,12,17 physician satisfaction,18 patient satisfaction,18 and many others. These outcomes are all components of value, but all are limited. It would be a mistake to make a general conclusion about the value of HIT by looking at only 1 of these measures. To make more general assessments, evaluations of HIT should take into account 3 principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder’s perspective is used.

Principle 1: Value Includes Both Costs and Benefits

Value, by definition, includes both costs and benefits. Studies that report only costs (or proxies for cost, such as utilization) cannot be used to reach conclusions about value. For example, 1 study quantified the costs of implementing an electronic health record (EHR) system for a 5-physician practice, including direct financial expenditures and hours needed to implement.19 These estimates of cost are an important component of value to that physician practice, but cannot by themselves be considered an estimate of value, because they do not include an assessment of potential benefits, which could include reduced costs that may accrue downstream. Nor do they include health benefits to patients, which may accrue in the near or long term as a result of the HIT implementation.

In another example of the limited ability of costs-only studies to define value, a population-based clinical decision support system was shown to increase the use of outpatient services and total medical costs in the intervention group, whereas services and costs did not significantly change in the control group.20 Again, without an assessment of benefits, these results are not a complete assessment of value, and general conclusions should not be made.

As a third example, the United States as a whole spends far more on healthcare now than it did in 1950, in large part due to technological advances. A newborn in 1950 could expect to require $8000 worth of medical care over the course of his or her lifetime (in 2001 dollars discounted to birth), whereas the comparable figure for a newborn in 1990 was $45,000, a nearly 600% increase. Yet, as Cutler and McClellan argue, based on an analysis of 5 medical conditions (heart attacks, low birth weight babies, depression, cataracts, and breast cancer), despite the rapid rise in costs, the benefits of medical progress (in terms of quality-adjusted life years) more than justify the added costs ($7 in benefits for every $1 in increased spending for heart attack treatments, for example).21

While it may not be feasible to measure all the costs and all the benefits of a particular HIT functionality, studies that only measure one or the other cannot by themselves provide assessments about the value of the technology.

Principle 2: Value Accrues Over Time

Value accrues in various degrees over time. The costs of an EHR, for example, from the perspective of a medical group, may entail purchase costs for the hardware and software, installation expenses, and training fees. These are all short-term costs. In the longer term, the EHR may require varying levels of maintenance, replacement costs, and retraining. Costs in terms of time burden may also vary over time, as clinicians become more adept at using the user interface.

Benefits vary across time as well. From the perspective of a patient, in the near term, HIT may result in a reduction in errors. In the long term, HIT may facilitate many health benefits through clinical monitoring, reminders, and disease management and prevention. In many cases, HIT may have both short- and long- term benefits. A computerized alert prompting the measurement of fasting lipids in a 40-year-old male may set the patient on a healthier trajectory, which may accrue over decades in cardiovascular disease avoided.

It is not feasible for any particular study to capture all the effects (especially in the long term) of any particular HIT intervention. However, the time horizon should be long enough to reflect the differences between the benefits and costs of using the HIT intervention versus not using it.22

Principle 3: Value Depends on Which Stakeholder’s Perspective Is Used

There are many possible perspectives from which to view the value of HIT. For example, consider 2 physician group practices, located across the street from each other,

both of which have implemented the same HIT system. Through the use of physician-reminder prompts, the delivery of screening colonoscopies increased 2-fold among eligible patients for both practices. If one physician group practice were reimbursed under fee-for-service, the increase in services would be a source of revenue, and therefore an increase in value to the practice. If instead of fee-for-service the other physician group practice were paid by capitation, this increase in services would be a cost to the group (in the short term, at least) and therefore a decrease in value. Even if the benefits to patients were equal in the 2 practices, the perception of value to the practices would be very different.

As another example, from the perspective of a small provider operating under fee-for-service, an EHR may necessitate that the physicians see fewer patients because of the increased documentation burden. But from a patient’s perspective, the improved documentation may result in better clinical decision making and improved health outcomes. From another patient’s perspective, however, the decrease in visit volume may have a negative effect in that it may make it more difficult for him or her to book a visit. From the payer’s perspective in this example, there may be a decrease in charges from the provider’s office for a period of time during the transition to an EHR, saving the payer money. However, more expensive emergency care may end up substituting for some of those visits, in which case the costs may increase, at least in the short term.

Thus, in any application of HIT, there may be “winners” and “losers” in terms of certain aspects of costs or utilization. In the Veterans Health Administration (VHA), for example, it takes more time for the physicians to enter their progress notes into the electronic health record than it did when paper was used, so the physicians “lose” in terms of time. But the VHA “wins” via reductions in the costs of storing and retrieving all the paper-based records, and the patients “win” by getting more preventive care services delivered to them, because the EHR contains decision support that prompts providers to deliver these services to eligible people. In some situations, the “winner” and “loser” is the same entity, depending on what is being looked at: physicians “lose” because it takes more time to enter information into the EHR, but they also “win” because every other physician has also had to enter their notes into the EHR, which means that all the time lost trying to figure out what happened at the last hospitalization or visit to the subspecialist disappears when this information is instantly available in the same EHR. These kinds of “transfers” can vastly influence conclusions about value, depending on the perspective.

Only 1 perspective is insensitive to such transfers when assessing the value of HIT, and it can be used as a summative evaluation for HIT as a whole: social value. Social value adds up all the costs paid directly or indirectly by patients (taxes, premiums through employers, premiums directly paid, direct payments as co-pays or self-pay) and benefits to patients (convenient access to providers, better health, longer lives). Any complete assessment of value of HIT for society must be from this social perspective.23

Social value does not always align with the business value to providers. If a physician group or hospital operates more efficiently and saves money as a result of HIT but the quality of the healthcare it delivers is unchanged, the social value of healthcare is also unchanged. However, if market forces result in healthcare organizations lowering the costs they charge payers, and if the payer passes on those savings to patients through decreased premiums, the social value will increase as a result of HIT. Unfortunately, market forces are imperfect in healthcare, and cost savings often do not get passed back to the patient, or if they do, it takes many years. It is therefore important for studies of HIT value to state how they expect the market to respond to the effect of the technology, including the time frame in which these changes may be reasonably expected to occur.

Ideally, all evaluations of HIT would take the perspective of all relevant stakeholders, including all possible costs and benefits, and consider the downstream effects of the technology in future decades. Clearly, such studies would be infeasible. However, we believe that authors of studies could do a better job in specifying which stakeholder’s perspective they are using as the lens for their evaluation, and in explaining the link to social value.


Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up