Currently Viewing:
The American Journal of Managed Care December 2016
Getting From Here to There: Health IT Needs for Population Health
Joshua R. Vest, PhD, MPH; Christopher A. Harle, PhD; Titus Schleyer, DMD, PhD; Brian E. Dixon, MPA, PhD, FHIMSS; Shaun J. Grannis, MD, MS, FAAFP, FACMI; Paul K. Halverson, DrPH, FACHE; and Nir Menachemi, PhD, MPH
The Health Information Technology Special Issue: Current Trends and Future Directions
Joshua R. Vest, PhD, MPH
How Health Plans Promote Health IT to Improve Behavioral Health Care
Amity E. Quinn, PhD; Sharon Reif, PhD; Brooke Evans, MA, MSW; Timothy B. Creedon, MA; Maureen T. Stewart, PhD; Deborah W. Garnick, ScD; and Constance M. Horgan, ScD
Data-Driven Clinical and Cost Pathways for Chronic Care Delivery
Yiye Zhang, PhD, and Rema Padman, PhD
Accountable Care Organization Hospitals Differ in Health IT Capabilities
Daniel M. Walker, PhD, MPH; Arthur M. Mora, PhD, MHA; and Ann Scheck McAlearney, ScD, MS
Building Health IT Capacity to Improve HIV Infection Health Outcomes
Hannah Rettler, MPH; R. Monina Klevens, DDS, MPH; Gillian Haney, MPH; Liisa Randall, PhD; Alfred DeMaria, MD; and Johanna Goderre, MPH
Telemedicine and the Sharing Economy: The "Uber" for Healthcare
Brian J. Miller, MD, MBA, MPH; Derek W. Moore, JD; and Chester W. Schmidt, Jr, MD
Currently Reading
Assessing Electronic Health Record Implementation Challenges Using Item Response Theory
Kitty S. Chan, PhD; Hadi Kharrazi, MD, PhD; Megha A. Parikh, MS; and Eric W. Ford, PhD, MPH
US Hospital Engagement in Core Domains of Interoperability
A. Jay Holmgren, BA; Vaishali Patel, PhD; Dustin Charles, MPH; and Julia Adler-Milstein, PhD

Assessing Electronic Health Record Implementation Challenges Using Item Response Theory

Kitty S. Chan, PhD; Hadi Kharrazi, MD, PhD; Megha A. Parikh, MS; and Eric W. Ford, PhD, MPH
It is unclear which barriers cause the greatest threats to the successful implementation of an electronic health record (EHR). This paper prioritizes the potential threats to EHR adoption using a novel analytic strategy: item response theory.

Objectives: To assess the importance of commonly identified issues in electronic health record (EHR) implementation using item response theory (IRT).

Study Design: Secondary data from the 2012 American Hospital Association’s Annual Survey Information Technology Supplement were used in the analyses. Results were compared and contrasted with the standard descriptive statistic frequencies that have been used to guide most recommendations made using the same data.

Methods: IRT was used to measure the magnitude of difficulty that particular challenges pose in implementing EHRs that meet federal guidelines for Meaningful Use.

Results: The IRT analyses yielded significantly different results from descriptive statistics in estimating the magnitude of specific EHR implementation challenges. In particular, IRT revealed that “obtaining physician cooperation” and “ongoing costs of maintaining and upgrading systems” were the most challenging implementation features. However, the frequency counts identified “upfront capital costs” and “complexity of meeting Meaningful Use criteria within implementation timeline” as the most challenging implementation features.

Conclusions: For managers and policy makers, having an accurate assessment of EHR implementation challenges is essential to designing effective programs. IRT provides a statistical approach that allows prior studies to be assessed more accurately and future studies to retain the easier-to-use, check-all-that-apply survey structure while gaining valuable information.

Am J Manag Care. 2016;22(12):e409-e415
Take-Away Points

Many surveys of hospital administrators have sought to identify barriers to the Meaningful Use of electronic health record (EHR) technology. However, the surveys used often lack the precision to provide a list of barriers ordered from most difficult to least difficult. Instead, surveys tend to report the most common or well-known barriers to adoption. 
  • The analysis identifies obtaining physician cooperation as the most challenging barrier to EHR adoption. 
  • Some frequently cited barriers, such as dealing with the complexity of the Meaningful Use requirements are shown to be less challenging.
Implementing an electronic health record (EHR) in a hospital brings many challenges—some bigger than others.1 Managers often recognize common but readily addressed EHR implementation issues, while more daunting challenges are identified less frequently or later in the process. Prior research has shown that hospitals are slow to implement key EHR features that enhance patient safety and care quality.2,3 In particular, professional buy-in and implementation costs have been identified as major impediments to EHR use.4 The latter issue—implementation cost barriers—is further exacerbated because demonstrating an EHR’s return on investment is difficult.5 Determining which EHR implementation issues hinder the widespread “Meaningful Use” of the technology is important.6

The problem in identifying and prioritizing EHR implementation issues stems from managerial knowledge gaps arising from a lack of experience with the process. To identify knowledge gaps, professional organizations survey health system leaders about their EHR implementation challenges.7,8 The American Hospital Association (AHA)’s Annual Survey Information Technology (IT) Supplement (referred to hereafter as “AHA Survey”) asks managers to pick among a list of options for all EHR implementation challenges that their hospital currently faces or anticipates. Although this approach is likely to capture how common an issue is among hospitals, it does not explicitly address how difficult a specific challenge is. Moreover, using response frequencies as a proxy measure for EHR implementation difficulty may lead to inaccurate inferences about the areas where managers should focus their efforts.

The purpose of this paper is 2-fold: first, the challenges hospital administrators face in implementing an EHR were examined using a holistic approach to item difficulty; second, an established psychometric methodology was employed to quantify the relative difficulty levels of various EHR implementation challenges, based on managers’ responses to a “pick all that apply” questionnaire. Item response theory (IRT)9 was used to analyze the AHA Survey’s questions related to EHR implementation challenges in meeting federal requirements for Meaningful Use.10 Results were compared and contrasted with the standard, descriptive statistics that have been widely reported.


A 2014 literature review of hospital EHR implementation studies identified 6 main categories of barriers11: 1) financial capabilities, related to the total cost of EHR system ownership (acquisition and operation); 2) vendor selection, related to integrating ITs into the workplace; 3) IT staffing, related to having employees with prior implementation experience; 4) health system culture, related to organizational collaboration challenges; 5) organizational complexity, related to the bureaucracy and change management capacity of the hospital; and 6) clinical professional barriers, related to the ITs’ impact on providers’ abilities to deliver care. Many dimensions play roles in the EHR implementation process, but 2 themes emerge consistently across the 6 domains identified in the prior research: experience with EHR implementation and the complexity of managing the particular barrier.

The common factors that affect a barrier’s potential importance can be put into a 2-dimensional framework of barrier complexity and experience. The experience dimension can be translated into where a health system is in its tacit knowledge about EHR implementation barriers. IRT is specifically designed to order respondents’ relative knowledge about specific content. The model is presented in the Figure.

The most complex managerial challenge when implementing an EHR is gaining clinician buy-in.12 By both social and legal conventions, physicians have the professional autonomy to make clinical decisions.13 The introduction of an EHR with predefined order sets, automated reminders, and decision support is perceived by many physicians as a direct attempt to diminish their professional autonomy. In addition, many physicians are not directly employed by health systems, further reducing their organizational commitment. Taken together, the resistance to managerial oversight and lack of organizational identification make gaining physician buy-in a major issue. Nurses have a lesser degree of professional autonomy and are typically employed by the hospital; hence, they tend to be less of a management challenge than physicians when it comes to EHR implementation and use. Nevertheless, the physician–nurse relationship is critical to EHR implementation success.

In the Figure, the clinical staff response (items 3 and 4) begins during the EHR adoption phase. Normatively, doctors and nurses should be brought in at the earliest stage. In practice, they tend to be included after the decision to adopt the EHR has been made during vendor selection.14 The importance of clinical staff engagement is an ongoing management challenge, as the degree of clinical functionality in the EHR and required care process documentation is ever-increasing. Therefore, management experience with implementing more sophisticated functionalities is a learned capability that continues ad infinitum.

The financial capacity (items 1 and 2) of the health system to support the EHR is also an ongoing challenge. The initial cost of purchasing an EHR is considerable, but knowable; however, it is the ongoing cost of operating the systems that many managers underestimate.15 Losses in productivity have a concomitant impact on revenues,16 and the double impact of unexpectedly high ownership costs and reduced revenue cause managers significant challenges. Managers typically become fully cognizant of these total cost of ownership (TCO) issues only after they begin the implementation process; the start date in the Figure reflects this fact.

Similar to the experience-informed learning that occurs with the recognition of TCO issues, the additional levels of organizational complexity (item 10) that are part-and-parcel of an EHR implementation become clearer over time.17 Post-EHR implementation, health systems have to roll out additional functionalities to meet organizational needs and comply with ever-increasing regulations. Hence, this construct also encompasses the entire EHR lifecycle.

Organizations’ cultures (item 9) are also impacted by the introduction of new technologies and the workflow changes that come with them.14 Depending on when organizational members are engaged in the EHR adoption and implementation processes, the impact on organizational culture can start very early on and continue indefinitely. One means of making the cultural shift a positive one is to engage staff members during the vendor selection phase to promote buy-in.

Vendor selection and operations are closely related to issues of EHR product certification (item 6), ensuring system security against privacy breaches (item 5), and ongoing system support (item 7). A major criterion in vendor selection is to reduce the management burden related to these items.18,19 Therefore, they appear lower on the management complexity axis. Despite being lower on the complexity dimension, ongoing vendor relations mean that it is an enduring management activity.

The lowest EHR implementation challenge construct, in terms of management complexity, is information technology staffing (item 8).14 The employees dedicated to supporting the technology are easier to manage because, unlike that of clinical staff, their job description explicitly includes following the direction of leadership. Nevertheless, managers often underestimate the number of IT staff needed to support an EHR.20 Therefore, the management challenge continues well into the implementation phase until the organization reaches some level of homeostasis with respect to the EHR.

The EHR adoption/implementation challenges used in the AHA Survey are not a perfect fit to the constructs listed in the literature review of Boonstra and colleagues.11 Nevertheless, the AHA Survey does touch on each of the major domains in some fashion and serves as a good starting point from which to explore EHR implementation challenges from a health system perspective.


The AHA Survey (2013)20 was funded by the Office of the National Coordinator for Health Information Technology (ONC) and sent to 5756 US hospitals (including those not members of the AHA). The survey had a 59% response rate for a sample of 3396. The survey question, “What is (are)/would be the primary challenge(s) in implementing an EHR system that meets the federal requirements for meaningful use? (Please check all that apply)” was analyzed. The item offered 10 structured responses, including: 1) upfront capital costs/lack of access to capital to install systems, 2) ongoing cost of maintaining and upgrading systems, 3) obtaining physician cooperation, 4) obtaining other staff cooperation, 5) concerns about security or liability for privacy breaches, 6) uncertainty about certification requirements, 7) limited vendor capacity, 8) lack of adequate IT personnel in hospital to support implementation/maintenance, 9) challenge/complexity of meeting all Meaningful Use criteria within implementation timeframe, and 10) complexity associated with coordinating decisions with system-level leadership. These responses were used in the IRT phase.

IRT is a collection of modeling techniques for analyzing item-level data obtained to measure variation between respondents. Unlike regression methods, IRT does not require a dependent variable against which to parse variance; instead, it compares the items against one another to establish the pattern of responses from a sample of survey or test takers. In addition, by evaluating the confidence intervals (CIs) of the item difficulty measurements, it is possible to assess if there are potentially missing items in the scale. When sorted by item difficulty, each CI should overlap with the next item in the list. CIs that do not overlap indicate that there may be another item that would improve the classification of survey respondents. The CI comparison can be used to detect potential gaps in an assessment scale’s items.

One important assumption of IRT is that the scale being assessed is unidimensional. To test this assumption, a factor analysis is performed in the items to ensure they are measuring a common construct (see eAppendix A [eAppendices available at] for a description of the factor analysis). Following that procedure, the IRT itself can be undertaken to assess both item and respondent characteristics (see eAppendix B for a description of the IRT).


Copyright AJMC 2006-2018 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up

Sign In

Not a member? Sign up now!