Currently Viewing:
The American Journal of Managed Care January 2018
Currently Reading
Measuring Overuse With Electronic Health Records Data
Thomas Isaac, MD, MBA, MPH; Meredith B. Rosenthal, PhD; Carrie H. Colla, PhD; Nancy E. Morden, MD, MPH; Alexander J. Mainor, JD, MPH; Zhonghe Li, MS; Kevin H. Nguyen, MS; Elizabeth A. Kinsella, BA; and Thomas D. Sequist, MD, MPH
Bridging the Digital Divide: Mobile Access to Personal Health Records Among Patients With Diabetes
Ilana Graetz, PhD; Jie Huang, PhD; Richard J. Brand, PhD; John Hsu, MD, MBA, MSCE; Cyrus K. Yamin, MD; and Mary E. Reed, DrPH
Electronic Health Record "Super-Users" and "Under-Users" in Ambulatory Care Practices
Juliet Rumball-Smith, MBChB, PhD; Paul Shekelle, MD, PhD; and Cheryl L. Damberg, PhD
Electronic Sharing of Diagnostic Information and Patient Outcomes
Darwyyn Deyo, PhD; Amir Khaliq, PhD; David Mitchell, PhD; and Danny R. Hughes, PhD
Hospital Participation in Meaningful Use and Racial Disparities in Readmissions
Mark Aaron Unruh, PhD; Hye-Young Jung, PhD; Rainu Kaushal, MD, MPH; and Joshua R. Vest, PhD, MPH
A Cost-Effectiveness Analysis of Cardiology eConsults for Medicaid Patients
Daren Anderson, MD; Victor Villagra, MD; Emil N. Coman, PhD; Ianita Zlateva, MPH; Alex Hutchinson, MBA; Jose Villagra, BS; and J. Nwando Olayiwola, MD, MPH
Electronic Health Record Problem Lists: Accurate Enough for Risk Adjustment?
Timothy J. Daskivich, MD, MSHPM; Garen Abedi, MD, MS; Sherrie H. Kaplan, PhD, MPH; Douglas Skarecky, BS; Thomas Ahlering, MD; Brennan Spiegel, MD, MSHS; Mark S. Litwin, MD, MPH; and Sheldon Greenfield, MD
Racial/Ethnic Variation in Devices Used to Access Patient Portals
Eva Chang, PhD, MPH; Katherine Blondon, MD, PhD; Courtney R. Lyles, PhD; Luesa Jordan, BA; and James D. Ralston, MD, MPH
Hospitalized Patients' and Family Members' Preferences for Real-Time, Transparent Access to Their Hospital Records
Michael J. Waxman, MD, MPH; Kurt Lozier, MBA; Lana Vasiljevic, MS; Kira Novakofski, PhD; James Desemone, MD; John O'Kane, RRT-NPS, MBA; Elizabeth M. Dufort, MD; David Wood, MBA; Ashar Ata, MBBS, PhD; Louis Filhour, PhD, RN; & Richard J. Blinkhorn Jr, MD

Measuring Overuse With Electronic Health Records Data

Thomas Isaac, MD, MBA, MPH; Meredith B. Rosenthal, PhD; Carrie H. Colla, PhD; Nancy E. Morden, MD, MPH; Alexander J. Mainor, JD, MPH; Zhonghe Li, MS; Kevin H. Nguyen, MS; Elizabeth A. Kinsella, BA; and Thomas D. Sequist, MD, MPH
Electronic health records data can accurately quantify overuse of clinical services and the risk factors that may trigger low-value testing and screening.

Objectives: To measure overuse of low-value care using electronic health record (EHR) data and manual chart review and to evaluate whether certain low-value services are better captured using EHR data.

Study Design: We implemented algorithms to extract performance on 13 Choosing Wisely–identified healthcare services using EHR data at a large physician practice group between 2011 and 2013.

Methods: We calculated rates of overuse using automated EHR extracts. We manually reviewed the charts for 200 cases of overuse for each measure to determine if they had clinical risk factors that could explain use of the low-value service and then calculated adjusted rates of overuse. We explored trends in overuse for each low-value service in the 3-year duration using logistic regression.

Results: Unadjusted rates of overuse ranged from 0.2% to 92%. Automated EHR extracts and manual chart review identified explanatory risk factors for most measures, although the magnitude varied: for some measures (eg, bone densitometry exam for women younger than 65 years), manual chart review did not identify many additional risks (3.0%). In contrast, in patients who had sinus computed tomography or an antibiotic prescription for uncomplicated acute rhinosinusitis, manual chart review identified more explanatory risk factors (22.5%) than the automated EHR extract (9.5%). Adjusted rates of overuse ranged from 0.2% to 61.9%. Eight services demonstrated a statistically significant decrease in overuse over 3 years, while 1 increased significantly.

Conclusions: The use of EHR data, both extracted and manually abstracted, provides an opportunity to more accurately and reliably identify overuse of low-value healthcare services.

Am J Manag Care. 2018;24(1):19-25
Takeaway Points
  • In 13 low-value tests and screenings, we found varying levels of overuse using both automatically extracted electronic health record (EHR) data and manual chart review. 
  • Although several studies reporting overuse of low-value health services rely on administrative claims data, we found that EHR data may accurately and reliably measure overuse. 
  • EHR data can provide important insights on the presence of clinical risk factors that may trigger or explain the use of low-value health services. 
  • EHR data extracts and manual chart review should be considered alongside other methodologies of measuring overuse to develop higher-value local treatment norms for clinicians.
In recent years, various stakeholders have sought strategies to reduce healthcare costs while improving the quality of care.1 One such strategy is to define, identify, and reduce wasteful spending on the delivery of unnecessary healthcare services. In 2012, the American Board of Internal Medicine launched the Choosing Wisely campaign, an initiative aimed at engaging patients and physicians in discussions regarding how to avoid unnecessary healthcare.2 More than 80 specialty societies have developed a list of recommendations to help physicians avoid unnecessary treatments, which include imaging studies, surgical procedures, medications, and laboratory testing.3

Findings of some early studies suggest that the Choosing Wisely initiative is showing modest results in its first years4,5; however, the success of this initiative, and others like it, is critically dependent on our ability to accurately measure healthcare overuse.6,7 Most studies in the literature have relied on administrative claims data,8-11 although these data are better suited for measuring simple processes of care and are not suitable to capture the nuances of appropriateness of care. Studies estimating overuse with administrative claims report that claims data can lack clinical context and that the use of such data risks misclassification of potentially appropriate testing or imaging.12-14 Electronic health records (EHRs) may capture more detailed clinical information and context behind inappropriate use that are not available in claims data.15-20 

The goal of our study was to measure overuse defined by the Choosing Wisely initiative using a combination of structured data extracts and manual chart review from EHRs. Using this combination of rich clinical information, we sought to determine whether certain types of low-value healthcare services are better captured than others using EHR data.


Study Setting

We conducted this study at Atrius Health, a large physician practice group with 900 physicians and more than 400 advanced practice clinicians that provides primary and specialty care for more than 740,000 patients in eastern Massachusetts. Atrius Health physicians have utilized an integrated EHR (Epic Systems) since 1995 to support computerized outpatient ordering of medications, laboratory tests, and radiologic studies. All outpatient encounters are entered into the medical record, including vital signs, clinical notes, diagnostic and procedure codes, and all laboratory and radiology results.

Choosing Wisely Recommendations

A team of 2 physician health services researchers and 2 health economists reviewed available Choosing Wisely recommendations for inclusion in the study. We selected 13 Choosing Wisely recommendations (Table 1) for analysis based on their relevance to both primary and specialty care; their inclusion of medications, procedures, and laboratory testing; and their focus on ambulatory care, given that Atrius Health does not provide hospital care and therefore does not capture these data reliably. 

We implemented algorithms to electronically capture performance on the selected Choosing Wisely recommendations between 2011 and 2013 using automated data extracts from coded data in the EHR. These data included the problem and medication lists and all diagnostic, lab, and procedure codes entered by clinicians during clinical encounters (eAppendix I [eAppendices available at]). The extracts implemented numerator and denominator inclusion definitions and captured any potential denominator exclusion criteria for each measure. We also used manual chart reviews to collect information on “explanatory risk factors” from the EHR, which were conditions that indicated why the Choosing Wisely test or treatment would have been clinically appropriate. Two measures (repeat bone densitometry [DEXA] exams and repeat endoscopy for Barrett’s esophagus) were assessed in years prior to 2011, as they required historic data to measure whether the repeat exams occurred between 2011 and 2013.

We measured overuse via a denominator at either the exam or patient level, based on whether we could identify a patient population accurately. Measures that defined the patient denominator based only on demographic criteria and required little clinical evaluation (eg, cancer screening measures for women) were measured using an exam-level denominator. In contrast, measures that defined the patient denominator based on presentation to the office with a symptom (eg, headache or syncope) were measured using a patient-level denominator. This approach allowed us to avoid making assumptions about whether patients were receiving their care at the physician group practice during the entire study period and should thus be included in the denominator. 

Manual Chart Review

From the electronic data extracts, we selected a random sample of 200 cases of overuse for each measure based on its numerator definition (eAppendix I) from each study year for manual chart review. Manual chart review data were considered the gold standard for our analyses, as such data are generally the most robust clinical information source available. The chart review team consisted of 2 board-certified internal medicine physicians and a research assistant. The manual chart reviews focused on the index clinical note associated with ordering of the targeted overuse service, as well as any other information referenced in this index clinical note, such as prior clinical notes, study results, or medication lists. 

The review team collected information on whether the EHR data extracts were working as intended in capturing the numerator, denominator, and exclusion definitions of each measure; developed a list of clinical explanatory risk factors for each measure based on guidelines; and examined whether these were present in the clinical notes or test ordering documentation. 


Data Analysis

We fit multivariable logistic regression models to assess trends of overuse using EHR data for each Choosing Wisely recommendation, incorporating definitions of overuse (eAppendix I) and controlling for each study year between 2009 and 2013.

The chart review team collected data on the number of explanatory factors found from EHR extracts and manual chart review. In cases where there was confusion about whether a test or procedure might have been clinically indicated, the 3 reviewers used a consensus approach to determine whether a legitimate explanatory risk factor was present.

We reported unadjusted and adjusted performance rates after excluding patients with explanatory risk factors. All analyses were conducted using Stata version 14 (StataCorp LP; College Station, Texas). The study protocol was approved by the Partners Healthcare System Human Studies Review Committee. 


The prevalence of overuse varied widely, both among those measures with exam-level denominators and those with patient-level denominators, after accounting for explanatory risk factors found in the EHR (Table 2). In 2013, among exam-level measures, the prevalence of overuse ranged from a low of 0.2% (proportion of Pap smears performed on women aged 18-21 years) to a high of 57% (proportion of DEXA exams performed on women aged 18-65 years). Among patient-level measures in 2013, the prevalence of overuse ranged from a low of 8% (use of head imaging for syncope) to a high of 92% (use of sinus computed tomography [CT] or antibiotics for acute rhinosinusitis).

Among the 12 measures with 3 years of data, the prevalence of overuse demonstrated a statistically significant decrease over time for 8 (67%) measures. There was a significant increase in prevalence for only 1 (8%) measure, the use of head imaging for patients with uncomplicated headache.

We found wide variation in the number of risk factors identified by the automated EHR extract across measures (Table 3). In Vitamin D deficiency screening, imaging for low-back pain, and both DEXA measures, over half of the sample was shown to have risk factors that could explain the test or screening using the automated EHR extract; however, other measures, like Pap smears performed on women younger than 21 years, found no explanatory risk factors when using the EHR extract alone. 

Across nearly all measures, the manual chart review identified additional explanatory risk factors, although the magnitude varied by measure (Table 3). Manual chart review identified few additional explanatory risk factors in DEXA exams performed on women younger than 65 years (3.0%) and prescription of opioids or butalbital treatment for patients with migraine (5.0%), but more in patients who had a sinus CT or antibiotic prescription for uncomplicated acute rhinosinusitis (22.5%) and Pap smears performed on women with total hysterectomy for noncancer disease (29.0%). eAppendix II presents the explanatory risk factors for each measure. 

We calculated an adjusted overuse rate in 2013 based on the proportion of EHR-identified overuse that manual chart review determined had risk factors that could explain the test or screening (Table 3). The prevalence of overuse after adjustment ranged between 0.2% for Pap smears on women younger than 21 years and 61.9% for patients with uncomplicated acute rhinosinusitis who had a sinus CT or were prescribed antibiotics. 


There is widespread agreement that low-value services are a substantial problem leading to enormous waste in healthcare, and Choosing Wisely recommendations are often suggested as a tool to begin to define and identify waste. We examined how easily EHR data from a large ambulatory care center could be used to identify overuse on a selection of Choosing Wisely measures, and we found mixed results. 

Previous study findings indicate that administrative claims are a useful, albeit limited, source of data for identifying overuse.14 Primarily used for documenting diagnoses and procedures provided for the purpose of payment, administrative claims data do not provide detailed clinical context, which can ultimately misclassify, underestimate, or overestimate indicated tests or screenings.21 Reliance on claims alone may misclassify a clinically appropriate test or screening as overuse of low-value care, as patient history is an integral factor in clinical decision making.22 Several studies, including our own, cite reliance solely on administrative claims data for measuring overuse as a limitation for accurately reporting overuse.15,23,24 

Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up