Currently Viewing:
The American Journal of Managed Care Special Issue: Health Information Technology
Improving Adherence to Cardiovascular Disease Medications With Information Technology
William M. Vollmer, PhD; Ashli A. Owen-Smith, PhD; Jeffrey O. Tom, MD, MS; Reesa Laws, BS; Diane G. Ditmer, PharmD; David H. Smith, PhD; Amy C. Waterbury, MPH; Jennifer L. Schneider, MPH; Cyndee H. Yonehara, BS; Andrew Williams, PhD; Suma Vupputuri, PhD; and Cynthia S. Rand, PhD
Information Retrieval Pathways for Health Information Exchange in Multiple Care Settings
Patrick Kierkegaard, PhD; Rainu Kaushal, MD, MPH; and Joshua R. Vest, PhD, MPH
The 3 Key Themes in Health Information Technology
Julia Adler-Milstein, PhD
Leveraging EHRs to Improve Hospital Performance: The Role of Management
Julia Adler-Milstein, PhD; Kirstin Woody Scott, MPhil; and Ashish K. Jha, MD, MPH
Electronic Alerts and Clinician Turnover: The Influence of User Acceptance
Sylvia J. Hysong, PhD; Christiane Spitzmuller, PhD; Donna Espadas, BS; Dean F. Sittig, PhD; and Hardeep Singh, MD, MPH
Cost Implications of Human and Automated Follow-up in Ambulatory Care
Eta S. Berner, EdD; Jeffrey H. Burkhardt, PhD; Anantachai Panjamapirom, PhD; and Midge N. Ray, MSN, RN
Primary Care Capacity as Insurance Coverage Expands: Examining the Role of Health Information Technology
Renuka Tipirneni, MD, MSc; Ezinne G. Ndukwe, MPH; Melissa Riba, MS; HwaJung Choi, PhD; Regina Royan, MPH; Danielle Young, MPH; Marianne Udow-Phillips, MHSA; and Matthew M. Davis, MD, MAPP
Adoption of Electronic Prescribing for Controlled Substances Among Providers and Pharmacies
Meghan Hufstader Gabriel, PhD; Yi Yang, MD, PhD; Varun Vaidya, PhD; and Tricia Lee Wilkins, PharmD, PhD
Health Information Exchange and the Frequency of Repeat Medical Imaging
Joshua R. Vest, PhD, MPH; Rainu Kaushal, MD, MPH; Michael D. Silver, MS; Keith Hentel, MD, MS; and Lisa M. Kern, MD
Information Technology and Hospital Patient Safety: A Cross-Sectional Study of US Acute Care Hospitals
Ajit Appari, PhD; M. Eric Johnson, PhD; and Denise L. Anthony, PhD
Automated Detection of Retinal Disease
Lorens A. Helmchen, PhD; Harold P. Lehmann, MD, PhD; and Michael D. Abràmoff, MD, PhD
Trending Health Information Technology Adoption Among New York Nursing Homes
Erika L. Abramson, MD, MS; Alison Edwards, MS; Michael Silver, MS; Rainu Kaushal, MD, MPH; and the HITEC investigators
Electronic Health Record Availability Among Advanced Practice Registered Nurses and Physicians
Janet M. Coffman, PhD, MPP, MA; Joanne Spetz, PhD; Kevin Grumbach, MD; Margaret Fix, MPH; and Andrew B. Bindman, MD
Currently Reading
The Value of Health Information Technology: Filling the Knowledge Gap
Robert S. Rudin, PhD; Spencer S. Jones, PhD; Paul Shekelle, MD, PhD; Richard J. Hillestad, PhD; and Emmett B. Keeler, PhD
The Effects of Health Information Technology Adoption and Hospital-Physician Integration on Hospital Efficiency
Na-Eun Cho, PhD; Jongwha Chang, PhD; and Bebonchu Atems, PhD

The Value of Health Information Technology: Filling the Knowledge Gap

Robert S. Rudin, PhD; Spencer S. Jones, PhD; Paul Shekelle, MD, PhD; Richard J. Hillestad, PhD; and Emmett B. Keeler, PhD
Studies of health information technology have not kept up with the evolving needs of the healthcare system. We explain how to set them straight.
Unfortunately, we have found that few studies include both costs and benefits in their definitions of value. Most studies look at only short-term time horizons, which ignore many of the downstream benefits of the HIT, and many studies don’t even explicitly state to whom the value is accruing. To illustrate this, we present in Table 1 a sample of recent HIT articles, and whether and how they reported information on value (columns A through D). These articles are a purposive sample to capture a diversity of study types, including those that focus on 1 particular HIT installation as well as broader cross-sectional studies.

In this sample of 10 articles, 3 reported only on costs, without mention of any benefits, while 1 reported only on benefits, with no reporting on costs (which can include utilization). Thus, 4 of the 10 studies do not report both costs and benefits, and therefore cannot provide information about value (since value includes both costs and benefits). All of the studies are relatively short term, most having a time horizon of 1 to 3 years, and thus any downstream benefits or costs that might accrue due to better care today will not be captured. Only 1 study was explicit about “To whom does the value accrue?”, although in 3 others this could be inferred. These studies clearly show room for improvement in terms of how they measure value.

Even when information on costs, benefits, and perspective are presented, studies of HIT must then present sufficient information for readers to know what the intervention is, if readers are to be able to reach conclusions about whether the HIT system is something that could be implemented in another context. However, most evaluations of HIT do not adequately report such information. Few studies explain how the technology works, let alone how it was implemented. Furthermore, most studies don’t even report basic contextual information such as under what payment system the users were paid.

Studies need to open up the “black box” of HIT and do a better job explaining how HIT contributed to the results, including the role of key contextual factors. The need for particulars about the intervention details and implementation details has been noted by others for several years.4,34,35 All of these authorities recommend including at least 3 descriptors of the intervention: what the functionality does, contextual information, and implementation information. HIT evaluation studies need to describe this information so that a reader can understand what was being evaluated and assess whether or not they might be able to realize that value if they were to try to implement the same intervention in a different context.

To show how studies currently achieve this benchmark, we assessed the same 10 studies as above along these dimensions (Table 1, columns E, F, and G). All the studies had some description of the HIT functionalities, although for some this was at a very general and basic level (“a hospital EHR with CPOE [computerized physician order entry],” otherwise unspecified). Four studies report no contextual information at all, while in another 3, the only contextual data are the size of the organizations (usually a hospital). Only 3 studies reported some aspect of the financial status or payer mix. In 8 studies, there was no information about implementation. These data support, broadly, conclusions similar to those reached in a recent systematic review of HIT covering the years 2010-2013. In that review of 225 studies, only about a third of studies reported any information about context and implementation beyond the simplest measures.6


Without more attention to the necessary measuring and reporting of the data needed to assess value, we risk the possibility of 3 more years’ worth of published studies, which we estimate would be more than 300 hypothesis-testing articles,6 that do not give us appreciably better knowledge about this crucial aspect of HIT: how best to realize value. We propose in Table 2 criteria that should be met by all reports of HIT studies that are intended to improve our understanding of value. These include using evaluations to include both costs and benefits, using longer time horizons, considering the perspectives of different stakeholders, and reporting implementation details and contextual variables.

Making these changes may require especially imaginative study designs, and research teams that have both quantitative and qualitative expertise. For example, more qualitative research will be required to assess such questions as: How were users of the HIT trained? Was the training likely effective? What kind of support was provided from organizational leadership? What exactly was the HIT product intended to do? To what extent were the end users involved in the design of the technology? Details like these are critical to determining the applicability of any particular study’s results to other settings. While an extensive body of qualitative literature shows how HIT may achieve benefits or result in unintended consequences, this literature is almost completely distinct from studies that assess value in terms of quantitative measures of efficiency, quality, and safety. It is therefore largely impossible, using the current literature, to both identify the successful implementations of HIT and understand how to replicate them.

Making such changes will not be easy, as ideas about what constitutes HIT research are entrenched. However, if research is to align with the current needs of the healthcare system, researchers should take on this challenge and produce results that not only prove what is possible, but also show how to achieve it.

Author Affiliations: RAND Health, Santa Monica, CA (PS, RJH, EBK) and Boston, MA (RSR, SSJ); Harvard Medical School, Division of General Internal Medicine, Brigham and Women’s Hospital (SSJ), Boston, MA; West Los Angeles VA Medical Center (PS), Los Angeles, CA; Vanguard Health Systems (SSJ), Nashville, TN.

Source of Funding: This work was funded by the Office of the National Coordinator for Health Information Technology under contract number HHSP23320095649WC.

Author Disclosure: Dr Shekelle is paid royalties from UpToDate, is a member of the National Guideline Clearinghouse editorial board, and is employed by the Department of Veterans Affairs. Dr Jones is now employed by Tenet Healthcare, which is in a contract relationship with Cerner Corp, which provides the organization's electronic health record system. Drs Rudin, Keeler, and Hillestad report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (RSR, SSJ, PS, RJH, EBK); acquisition of data (SSJ, PS); analysis and interpretation of data (SSJ, PS); drafting of the manuscript (RSR, SSJ, PS, RJH); critical revision of the manuscript for important intellectual content (RSR, SSJ, PS, RJH, EBK); obtaining funding (SSJ, PS); supervision (RSR, PS).

Address correspondence to: Robert S. Rudin, PhD, RAND Health, 20 Park Plaza, Ste 920, Boston, MA 02116. E-mail:
1. Hillestad R, Bigelow J, Bower A, et al. Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health Aff (Millwood). 2005;24(5):1103-1117.

2. Walker J, Pan E, Johnston D, Adler-Milstein J, Bates DW, Middleton B. The value of health care information exchange and interoperability. Health Aff (Millwood). 2005;Suppl Web Exclusives:W5-10-W5-8.

3. Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011;30(3):464-471.

4. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-752.

5. Goldzweig CL, Towfigh A, Maglione M, Shekelle PG. Costs and benefits of health information technology: new trends from the literature. Health Aff (Millwood). 2009;28(2):w282-w293.

6. Shekelle PG, Jones SS, Rudin RS, et al, of the Southern California Evidence-based Practice Center of the RAND Corporation. Health Information Technology: An Updated Systematic Review with a Focus on Meaningful Use Functionalities. Washington, DC: Office of the National Coordinator for Health Information Technology; 2014.

7. Pettit RS, Zimmerman CR, Alaniz C, Dorsch MP. Cost analysis before and after implementation of a computerized physician order entry order form for enoxaparin. PT. 2012;37(2):107-111.

8. Devine EB, Hollingworth W, Hansen RN, et al. Electronic prescribing at the point of care: a time-motion study in the primary care setting. Health Serv Res. 2010;45(1):152-171.

9. Duffy WJ, Kharasch MS, Du H. Point of care documentation impact on the nurse-patient interaction. Nurs Adm Q. 2010;34(1):E1-E10.

10. Furukawa MF. Electronic medical records and efficiency and productivity during office visits. Am J Manag Care. 2011;17(4):296-303.

11. Mäenpää T, Asikainen P, Gissler M, et al. Outcomes assessment of the regional health information exchange: a five-year follow-up study. Methods Inf Med. 2011;50(4):308-318.

12. Stenner SP, Chen Q, Johnson KB. Impact of generic substitution decision support on electronic prescribing behavior. J Am Med Inform Assoc. 2010;17(6):681-688.

13. Bell DS, Cima L, Seiden DS, Nakazono TT, Alcouloumre MS, Cunningham WE. Effects of laboratory data exchange in the care of patients with HIV. Int J Med Inform. 2012;81(10):e74-e82.

14. Herrin J, da Graca B, Nicewander D, et al. The effectiveness of implementing an electronic health record on diabetes care and outcomes. Health Serv Res. 2012;47(4):1522-1540.

15. Quinn CC, Shardell MD, Terrin ML, Barr EA, Ballew SH, Gruber-Baldini AL. Cluster-randomized trial of a mobile phone personalized behavioral intervention for blood glucose control. Diabetes Care. 2011;34(9):1934-1942.

16. Gustafson D, Wise M, Bhattacharya A, et al. The effects of combining Web-based eHealth with telephone nurse case management for pediatric asthma control: a randomized controlled trial. J Med Internet Res. 2012;14(4):e101.

17. Abramson EL, Barrón Y, Quaresimo J, Kaushal R. Electronic prescribing within an electronic health record reduces ambulatory prescribing errors. Jt Comm J Qual Patient Saf. 2011;37(10):470-478.

18. Duffy RL, Yiu SS, Molokhia E, Walker R, Perkins RA. Effects of electronic prescribing on the clinical practice of a family medicine residency. Fam Med. 2010;42(5):358-363.

19. Fleming NS, Culler SD, McCorkle R, Becker ER, Ballard DJ. The financial and nonfinancial costs of implementing electronic health records in primary care practices. Health Aff (Millwood). 2011;30(3):481-489.

20. Eisenstein EL, Anstrom KJ, Edwards R, Willis JM, Simo J, Lobach DF. Population-based clinical decision support: a clinical and economic evaluation. Stud Health Technol Inform. 2012;180:343-347.

21. Cutler DM, McClellan M. Is technological change in medicine worth it? Health Affairs. 2001;20(5):11-29.

22. Weinstein MC, O’Brien B, Hornberger J, et al; ISPOR Task Force on Good Research Practices—Modeling Studies. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—Modeling Studies. Value Health. 2003;6(1):9-17.

23. Gold M. Panel on cost-effectiveness in health and medicine. Med Care. 1996;34(12 suppl):DS197-DS199.

24. Connelly DP, Park YT, Du J, et al. The impact of electronic health records on care of heart failure patients in the emergency room. J Am Med Inform Assoc. 2012;19(3):334-340.

25. Furukawa MF, Raghu TS, Shao BB. Electronic medical records, nurse staffing, and nurse-sensitive patient outcomes: evidence from California hospitals, 1998-2007. Health Serv Res. 2010;45(4):941-962.

26. Himmelstein DU, Wright A, Woolhandler S. Hospital computing and the costs and quality of care: a national study. Am J Med. 2010;123(1):40-46.

27. Lau AY, Sintchenko V, Crimmins J, Magrabi F, Gallego B, Coiera E. Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial. J Am Med Inform Assoc. 2012;19(5):719-727.

28. Ling SB, Richardson DB, Mettenbrink CJ, et al. Evaluating a web-based test results system at an urban STI clinic. Sex Transm Dis. 2010; 37(4):259-263.

29. McCormick D, Bor DH, Woolhandler S, Himmelstein DU. Giving office-based physicians electronic access to patients’ prior imaging and lab results did not deter ordering of tests. Health Aff (Millwood). 2012;31(3):488-496.

30. Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. JAMA. 2012;308(19):2012-2019.

31. Shapiro JS, Baumlin KM, Chawla N, et al. Emergency department information system implementation and process redesign result in rapid and sustained financial enhancement at a large academic center. Acad Emerg Med. 2010;17(5):527-535.

32. Virga PH, Jin B, Thomas J, Virodov S. Electronic health information technology as a tool for improving quality of care and health outcomes for HIV/AIDS patients. Int J Med Inform. 2012;81(10):e39-e45.

33. Zlabek JA, Wickus JW, Mathiason MA. Early cost and safety benefits of an inpatient electronic health record. J Am Med Inform Assoc. 2011;18(2):169-172.

34. Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261-1263.

35. Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI—statement on reporting of evaluation studies in Health Informatics. Int J Med Inform. 2009;78(1):1-9.
Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up