The Value of Health Information Technology: Filling the Knowledge Gap

Studies of health information technology have not kept up with the evolving needs of the healthcare system. We explain how to set them straight.
Published Online: January 26, 2015
Robert S. Rudin, PhD; Spencer S. Jones, PhD; Paul Shekelle, MD, PhD; Richard J. Hillestad, PhD; and Emmett B. Keeler, PhD
Despite rapid growth in the rate of adoption of health information technology (HIT), and in the volume of evaluation studies, the existing knowledge base for the value of HIT is not advancing at a similar rate. Most evaluation articles are limited in that they use incomplete measures of value and fail to report the important contextual and implementation characteristics that would allow for an adequate understanding of how the study results were achieved. To address these deficiencies, we present a conceptual framework for measuring HIT value and we propose a checklist of characteristics that should be considered in HIT evaluation studies. The framework consists of 3 key principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder’s perspective is used. Through examples, we show how these principles can be used to guide and improve HIT evaluation studies. The checklist includes a list of contextual and implementation characteristics that are important for interpretation of results. These improvements will make future studies more useful for policy makers and more relevant to the current needs of the healthcare system.

Am J Manag Care. 2014;20(11 Spec No. 17):eSP1-eSP8
Healthcare organizations all over the United States are adopting health information technology (HIT), yet recent studies of HIT are often unhelpful to these organizations or to those making policy decisions. Most studies fail to use adequate measures of value or to report key contextual and implementation characteristics. In this commentary, we:
  • Demonstrate the problem in a sampling of published articles.
  • Propose a conceptual framework and principles for measuring value in HIT.
  • Present a checklist of key study characteristics.

Readers can use this article as a touchstone for assessing the utility and relevance of published research studies.
In 2010, the United States Congress enacted the Health Information Technology for Economic and Clinical Health (HITECH) Act, thereby dedicating roughly $27 billion to eligible healthcare providers who demonstrate “meaningful use” of health information technology (HIT). This action motivated the purchase and installation of tens of thousands of HIT products across the US healthcare system and placed a national spotlight on HIT. The HITECH Act was justified, in part, by early studies that documented positive effects on healthcare quality, safety, and efficiency at a subset  of institutions that were early to implement HIT, and by policy analyses that extrapolated the implications of these potential benefits throughout the US healthcare system.1,2 These studies suggested that HIT had the potential to create enormous value by facilitating improvements in the quality of healthcare and reductions in costs.

However, systematic reviews of HIT have found that the evidence for value is inconclusive and that existing studies suffer from major limitations.3-6 This finding is true even of the most recent literature reviews, despite a greatly increasing number of studies evaluating HIT.6 In this paper, we suggest a way to overcome these deficiencies and make HIT evaluations more relevant to the current needs of the healthcare system, by presenting a conceptual framework for measuring the value of HIT, examining how a sample of published HIT articles report the information needed to make meaningful assessments of value, and proposing a set of criteria for future evaluations that would make them more useful for policy makers.

This paper was sponsored by the Office of the National Coordinator for Health Information Technology, which otherwise had no role in the conduct or writing of the paper or the decision to publish it.


Published studies have evaluated HIT using many different measures, including cost savings,7 time savings,8,9 utilization reduction,10-12 process adherence improvements,13,14 health outcomes,15,16 safety improvements,12,17 physician satisfaction,18 patient satisfaction,18 and many others. These outcomes are all components of value, but all are limited. It would be a mistake to make a general conclusion about the value of HIT by looking at only 1 of these measures. To make more general assessments, evaluations of HIT should take into account 3 principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder’s perspective is used.

Principle 1: Value Includes Both Costs and Benefits

Value, by definition, includes both costs and benefits. Studies that report only costs (or proxies for cost, such as utilization) cannot be used to reach conclusions about value. For example, 1 study quantified the costs of implementing an electronic health record (EHR) system for a 5-physician practice, including direct financial expenditures and hours needed to implement.19 These estimates of cost are an important component of value to that physician practice, but cannot by themselves be considered an estimate of value, because they do not include an assessment of potential benefits, which could include reduced costs that may accrue downstream. Nor do they include health benefits to patients, which may accrue in the near or long term as a result of the HIT implementation.

In another example of the limited ability of costs-only studies to define value, a population-based clinical decision support system was shown to increase the use of outpatient services and total medical costs in the intervention group, whereas services and costs did not significantly change in the control group.20 Again, without an assessment of benefits, these results are not a complete assessment of value, and general conclusions should not be made.

As a third example, the United States as a whole spends far more on healthcare now than it did in 1950, in large part due to technological advances. A newborn in 1950 could expect to require $8000 worth of medical care over the course of his or her lifetime (in 2001 dollars discounted to birth), whereas the comparable figure for a newborn in 1990 was $45,000, a nearly 600% increase. Yet, as Cutler and McClellan argue, based on an analysis of 5 medical conditions (heart attacks, low birth weight babies, depression, cataracts, and breast cancer), despite the rapid rise in costs, the benefits of medical progress (in terms of quality-adjusted life years) more than justify the added costs ($7 in benefits for every $1 in increased spending for heart attack treatments, for example).21

While it may not be feasible to measure all the costs and all the benefits of a particular HIT functionality, studies that only measure one or the other cannot by themselves provide assessments about the value of the technology.

Principle 2: Value Accrues Over Time

Value accrues in various degrees over time. The costs of an EHR, for example, from the perspective of a medical group, may entail purchase costs for the hardware and software, installation expenses, and training fees. These are all short-term costs. In the longer term, the EHR may require varying levels of maintenance, replacement costs, and retraining. Costs in terms of time burden may also vary over time, as clinicians become more adept at using the user interface.

Benefits vary across time as well. From the perspective of a patient, in the near term, HIT may result in a reduction in errors. In the long term, HIT may facilitate many health benefits through clinical monitoring, reminders, and disease management and prevention. In many cases, HIT may have both short- and long- term benefits. A computerized alert prompting the measurement of fasting lipids in a 40-year-old male may set the patient on a healthier trajectory, which may accrue over decades in cardiovascular disease avoided.

It is not feasible for any particular study to capture all the effects (especially in the long term) of any particular HIT intervention. However, the time horizon should be long enough to reflect the differences between the benefits and costs of using the HIT intervention versus not using it.22

Principle 3: Value Depends on Which Stakeholder’s Perspective Is Used

There are many possible perspectives from which to view the value of HIT. For example, consider 2 physician group practices, located across the street from each other,

both of which have implemented the same HIT system. Through the use of physician-reminder prompts, the delivery of screening colonoscopies increased 2-fold among eligible patients for both practices. If one physician group practice were reimbursed under fee-for-service, the increase in services would be a source of revenue, and therefore an increase in value to the practice. If instead of fee-for-service the other physician group practice were paid by capitation, this increase in services would be a cost to the group (in the short term, at least) and therefore a decrease in value. Even if the benefits to patients were equal in the 2 practices, the perception of value to the practices would be very different.

As another example, from the perspective of a small provider operating under fee-for-service, an EHR may necessitate that the physicians see fewer patients because of the increased documentation burden. But from a patient’s perspective, the improved documentation may result in better clinical decision making and improved health outcomes. From another patient’s perspective, however, the decrease in visit volume may have a negative effect in that it may make it more difficult for him or her to book a visit. From the payer’s perspective in this example, there may be a decrease in charges from the provider’s office for a period of time during the transition to an EHR, saving the payer money. However, more expensive emergency care may end up substituting for some of those visits, in which case the costs may increase, at least in the short term.

Thus, in any application of HIT, there may be “winners” and “losers” in terms of certain aspects of costs or utilization. In the Veterans Health Administration (VHA), for example, it takes more time for the physicians to enter their progress notes into the electronic health record than it did when paper was used, so the physicians “lose” in terms of time. But the VHA “wins” via reductions in the costs of storing and retrieving all the paper-based records, and the patients “win” by getting more preventive care services delivered to them, because the EHR contains decision support that prompts providers to deliver these services to eligible people. In some situations, the “winner” and “loser” is the same entity, depending on what is being looked at: physicians “lose” because it takes more time to enter information into the EHR, but they also “win” because every other physician has also had to enter their notes into the EHR, which means that all the time lost trying to figure out what happened at the last hospitalization or visit to the subspecialist disappears when this information is instantly available in the same EHR. These kinds of “transfers” can vastly influence conclusions about value, depending on the perspective.

PDF is available on the last page.
Adult ADHD Compendium
COPD Compendium
Dermatology Compendium
Diabetes Compendium
Hematology Compendium
Immuno-oncology Compendium
Lipids Compendium
MACRA Compendium
Neutropenia Compendium
Oncology Compendium
Pain Compendium
Reimbursement Compendium
Rheumatoid Arthritis Compendium
Know Your News
HF Compendium
Managed Care PODCAST