Currently Viewing:
The American Journal of Managed Care January 2019
The Gamification of Healthcare: Emergence of the Digital Practitioner?
Eli G. Phillips Jr, PharmD, JD; Chadi Nabhan, MD, MBA; and Bruce A. Feinberg, DO
From the Editorial Board: Rajesh Balkrishnan, PhD
Rajesh Balkrishnan, PhD
The Health Information Technology Special Issue: New Real-World Evidence and Practical Lessons
Mary E. Reed, DrPH
Currently Reading
Inpatient Electronic Health Record Maintenance From 2010 to 2015
Vincent X. Liu, MD, MS; Nimah Haq, MPH; Ignatius C. Chan, MD; and Brian Hoberman, MD, MBA
Mind the Gap: The Potential of Alternative Health Information Exchange
Jordan Everson, PhD; and Dori A. Cross, PhD
Patient and Clinician Experiences With Telehealth for Patient Follow-up Care
Karen Donelan, ScD, EdM; Esteban A. Barreto, MA; Sarah Sossong, MPH; Carie Michael, SM; Juan J. Estrada, MSc, MBA; Adam B. Cohen, MD; Janet Wozniak, MD; and Lee H. Schwamm, MD
Understanding the Relationship Between Data Breaches and Hospital Advertising Expenditures
Sung J. Choi, PhD; and M. Eric Johnson, PhD
Organizational Influences on Healthcare System Adoption and Use of Advanced Health Information Technology Capabilities
Paul T. Norton, MPH, MBA; Hector P. Rodriguez, PhD, MPH; Stephen M. Shortell, PhD, MPH, MBA; and Valerie A. Lewis, PhD, MA
Alternative Payment Models and Hospital Engagement in Health Information Exchange
Sunny C. Lin, MS; John M. Hollingsworth, MD, MS; and Julia Adler-Milstein, PhD
Drivers of Health Information Exchange Use During Postacute Care Transitions
Dori A. Cross, PhD; Jeffrey S. McCullough, PhD; and Julia Adler-Milstein, PhD

Inpatient Electronic Health Record Maintenance From 2010 to 2015

Vincent X. Liu, MD, MS; Nimah Haq, MPH; Ignatius C. Chan, MD; and Brian Hoberman, MD, MBA
In the 6 years following inpatient electronic health record (EHR) implementation, an average of 2.5 significant EHR changes per day were made for maintenance and improvement.

Between 2010 and 2015, 5551 unique changes were made to the inpatient EHR (Figure 1), with a median of 72 (IQR, 35-112) changes per month. Most unique changes (n = 3191 [57.5%]) were updates to existing functionality, with 95.7% affecting all 21 hospitals. Individual changes were aggregated within 2190 update communication documents.

Upgrades related to EHR orders contributed to the largest proportion of all significant changes (44.7% of documents) (Figure 2). In total, changes to templated order sets comprised 29.9% of all documents. Other EHR functional domains that accounted for a significant proportion of all changes included clinical data review (15.7%), surgical and ED-specific tools (13.5%), alerts and customization (11.4%), grouped reports and HIM (8.3%), and patient tools and other (6.3%). Overall, changes affected 135 EHR functions.

In total, 151 specific types of users were affected by changes (eg, bed controller, cardiologist, certified nurse midwife, support site specialist), with an impact on all KPHC users 10.2% of the time. Targeted changes most frequently affected nurses (30.6%), physicians (26.6%), and other clinical staff (22.7%), such as pharmacists, therapists, and dietitians. The specific clinical areas most commonly affected by changes included surgical specialties (7.9%), ED (7.1%), mother–baby (6.9%), and pharmacy (6.4%) (Table).


Over a 6-year period, the changes required to maintain and improve an inpatient EHR system were substantial and diverse, with a pervasive impact. On average, 2.5 significant EHR changes occurred each day, together affecting more than 130 specific tools and 150 unique user roles across the 21 hospitals. Key areas that were frequently targeted by updates included specific EHR tools (order sets), clinical domains (surgical and ED), and end users (nurses, physicians, and pharmacists).

Widespread implementation of inpatient EHR systems has occurred rapidly over the past decade; however, few studies have comprehensively detailed the maintenance required to optimize their use.4,5 Most studies evaluating EHR implementation have focused on quantifying the costs and barriers related to initial implementation, the outcomes associated with EHR uptake, and the impact of EHR use on clinicians and patients.6-18,20-25 Nevertheless, emerging evidence suggests that the benefits of EHR use accrue gradually over time and are likely attributable to the ongoing addition of new functionality attained via continual updates and upgrades.21,26 Thus, although much attention is focused on the initial “go live” of the system, the true benefits of EHR adoption may only emerge with persistent attention to enhancing the EHR-based workflows and tools that drive improvements in care. In particular, the customization and usability of EHR functions to meet end-user needs have been identified as key measures that portend likely EHR benefit and can also mitigate potential harm arising from usability or workflow challenges.27-30


The primary limitation of this study is that our findings were based on a single healthcare system and a specific EHR product, which may limit the generalizability of our findings. Our study focused on the clinical aspects of inpatient EHR maintenance recorded within monthly change communication reports. However, our findings almost certainly represent a significant underestimate of the true scale and scope of ongoing EHR changes across our system. Numerous daily changes are made to EHR functions that do not rise to the significance level that would trigger their inclusion within communication reports. Our findings also do not account for simultaneous outpatient EHR and information technology infrastructure support, which contributes heavily to ongoing maintenance needs. Finally, the resources needed to implement each change could vary significantly, in terms of both time and cost.


EHR maintenance needs were prevalent and diverse, affecting 150 unique user roles and contributing to an average of more than 2.5 significant changes per day. Our findings highlight the need for significant resources, expertise, and collaboration to maximize EHR clinical utility and benefit. They also demonstrate that an EHR system represents a dynamic network of evolving tools that requires ongoing investment well after initial implementation.

Author Affiliations: Kaiser Permanente Division of Research (VXL), Oakland, CA; Master of Public Health Program, University of Southern California (NH), Los Angeles, CA; The Permanente Medical Group (VXL, ICC, BH), Oakland, CA.

Source of Funding: The Permanente Medical Group, National Institutes of Health/National Institute of General Medical Sciences (NIH/NIGMS) K23 GM112018, and NIH/NIGMS R35 GM128672.

Author Disclosures: Dr Liu is employed with The Permanente Medical Group and received the NIH/NIGMS K23 GM112018 grant. Dr Chan and Dr Hoberman are employed with The Permanente Medical Group. Ms Haq reports no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (VXL, NH, ICC, BH); acquisition of data (VXL, ICC, BH); analysis and interpretation of data (VXL, NH, ICC, BH); drafting of the manuscript (VXL); critical revision of the manuscript for important intellectual content (VXL, NH, ICC, BH); statistical analysis (VXL, NH); provision of patients or study materials (ICC); obtaining funding (VXL); administrative, technical, or logistic support (VXL, ICC, BH); and supervision (VXL, ICC, BH).

Address Correspondence to: Vincent X. Liu, MD, MS, Kaiser Permanente Division of Research, 2000 Broadway, Oakland, CA 95070. Email:

1. Non-federal acute care hospital electronic health record adoption. Office of the National Coordinator for Health Information Technology website. Updated September 2017. Accessed July 1, 2018.

2. Blumenthal D. Launching HITECH. N Engl J Med. 2010;362(5):382-385. doi: 10.1056/NEJMp0912825.

3. Adler-Milstein J, Jha AK. HITECH Act drove large gains in hospital electronic health record adoption. Health Aff (Millwood). 2017;36(8):1416-1422. doi: 10.1377/hlthaff.2016.1651.

4. Jha AK, DesRoches CM, Campbell EG, et al. Use of electronic health records in U.S. hospitals. N Engl J Med. 2009;360(16):1628-1638. doi: 10.1056/NEJMsa0900592.

5. Dorr DA, Cohen DJ, Adler-Milstein J. Data-driven diffusion of innovations: successes and challenges in 3 large-scale innovative delivery models. Health Aff (Millwood). 2018;37(2):257-265. doi: 10.1377/hlthaff.2017.1133.

6. Fleming NS, Culler SD, McCorkle R, Becker ER, Ballard DJ. The financial and nonfinancial costs of implementing electronic health records in primary care practices. Health Aff (Millwood). 2011;30(3):481-489. doi: 10.1377/hlthaff.2010.0768.

7. Wang SJ, Middleton B, Prosser LA, et al. A cost-benefit analysis of electronic medical records in primary care. Am J Med. 2003;114(5):397-403. doi: 10.1016/S0002-9343(03)00057-3.

8. Evans DC, Nichol WP, Perlin JB. Effect of the implementation of an enterprise-wide electronic health record on productivity in the Veterans Health Administration. Health Econ Policy Law. 2006;1(2):163-169. doi: 10.1017/S1744133105001210.

9. Gagnon MP, Nsangou ÉR, Payne-Gagnon J, Grenier S, Sicotte C. Barriers and facilitators to implementing electronic prescription: a systematic review of user groups’ perceptions. J Am Med Inform Assoc. 2014;21(3):535-541. doi: 10.1136/amiajnl-2013-002203.

10. Kruse CS, Kothman K, Anerobi K, Abanaka L. Adoption factors of the electronic health record: a systematic review. JMIR Med Inform. 2016;4(2):e19. doi: 10.2196/medinform.5525.

11. Kruse CS, Kristof C, Jones B, Mitchell E, Martinez A. Barriers to electronic health record adoption: a systematic literature review. J Med Syst. 2016;40(12):252. doi: 10.1007/s10916-016-0628-9.

12. McGinn CA, Grenier S, Duplantie J, et al. Comparison of user groups’ perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Med. 2011;9:46. doi: 10.1186/1741-7015-9-46.

13. Zandieh SO, Yoon-Flannery K, Kuperman GJ, Langsam DJ, Hyman D, Kaushal R. Challenges to EHR implementation in electronic- versus paper-based office practices. J Gen Intern Med. 2008;23(6):755-761. doi: 10.1007/s11606-008-0573-5.

14. Burton LC, Anderson GF, Kues IW. Using electronic health records to help coordinate care. Milbank Q. 2004;82(3):457-481, table of contents. doi: 10.1111/j.0887-378X.2004.00318.x.

15. Babbott S, Manwell LB, Brown R, et al. Electronic medical records and physician stress in primary care: results from the MEMO Study. J Am Med Inform Assoc. 2014;21(e1):e100-e106. doi: 10.1136/amiajnl-2013-001875.

16. Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res. 2010;10:231. doi: 10.1186/1472-6963-10-231.

17. Howley MJ, Chou EY, Hansen N, Dalrymple PW. The long-term financial impact of electronic health record implementation. J Am Med Inform Assoc. 2015;22(2):443-452. doi: 10.1136/amiajnl-2014-002686.

18. Adler-Milstein J, DesRoches CM, Kralovec P, et al. Electronic health record adoption in US hospitals: progress continues, but challenges persist. Health Aff (Millwood). 2015;34(12):2174-2180. doi: 10.1377/hlthaff.2015.0992.

19. Liang LL, ed. Connected for Health: Using Electronic Health Records to Transform Care Delivery. San Francisco, CA: Jossey-Bass; 2010.

20. Reed M, Huang J, Graetz I, et al. Outpatient electronic health records and the clinical care and outcomes of patients with diabetes mellitus. Ann Intern Med. 2012;157(7):482-489. doi: 10.7326/0003-4819-157-7-201210020-00004.

21. Lin SC, Jha AK, Adler-Milstein J. Electronic health records associated with lower hospital mortality after systems have time to mature. Health Aff (Millwood). 2018;37(7):1128-1135. doi: 10.1377/hlthaff.2017.1658.

22. King J, Patel V, Jamoom EW, Furukawa MF. Clinical benefits of electronic health record use: national findings. Health Serv Res. 2014;49(1, pt 2):392-404. doi: 10.1111/1475-6773.12135.

23. Walsh SH. The clinician’s perspective on electronic health records and how they can affect patient care. BMJ. 2004;328(7449):1184-1187. doi: 10.1136/bmj.328.7449.1184.

24. Hillestad R, Bigelow J, Bower A, et al. Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health Aff (Millwood). 2005;24(5):1103-1117. doi: 10.1377/hlthaff.24.5.1103.

25. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-752. doi: 10.7326/0003-4819-144-10-200605160-00125.

26. Adler-Milstein J, Everson J, Lee SY. EHR adoption and hospital performance: time-related effects. Health Serv Res. 2015;50(6):1751-1771. doi: 10.1111/1475-6773.12406.

27. Gelmon S, Bouranis N, Sandberg B, Petchel S. Strategies for addressing the challenges of patient-centered medical home implementation: lessons from Oregon. J Am Board Fam Med. 2018;31(3):334-341. doi: 10.3122/jabfm.2018.03.170265.

28. Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA. 2018;319(12):1276-1278. doi: 10.1001/jama.2018.1171.

29. Ellsworth MA, Dziadzko M, O’Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc. 2017;24(1):218-226. doi: 10.1093/jamia/ocw046.

30. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi: 10.1093/jamia/ocv050.
Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up