Currently Viewing:
The American Journal of Managed Care January 2018
Currently Reading
Measuring Overuse With Electronic Health Records Data
Thomas Isaac, MD, MBA, MPH; Meredith B. Rosenthal, PhD; Carrie H. Colla, PhD; Nancy E. Morden, MD, MPH; Alexander J. Mainor, JD, MPH; Zhonghe Li, MS; Kevin H. Nguyen, MS; Elizabeth A. Kinsella, BA; and Thomas D. Sequist, MD, MPH
Bridging the Digital Divide: Mobile Access to Personal Health Records Among Patients With Diabetes
Ilana Graetz, PhD; Jie Huang, PhD; Richard J. Brand, PhD; John Hsu, MD, MBA, MSCE; Cyrus K. Yamin, MD; and Mary E. Reed, DrPH
Electronic Health Record "Super-Users" and "Under-Users" in Ambulatory Care Practices
Juliet Rumball-Smith, MBChB, PhD; Paul Shekelle, MD, PhD; and Cheryl L. Damberg, PhD
Electronic Sharing of Diagnostic Information and Patient Outcomes
Darwyyn Deyo, PhD; Amir Khaliq, PhD; David Mitchell, PhD; and Danny R. Hughes, PhD
Hospital Participation in Meaningful Use and Racial Disparities in Readmissions
Mark Aaron Unruh, PhD; Hye-Young Jung, PhD; Rainu Kaushal, MD, MPH; and Joshua R. Vest, PhD, MPH
A Cost-Effectiveness Analysis of Cardiology eConsults for Medicaid Patients
Daren Anderson, MD; Victor Villagra, MD; Emil N. Coman, PhD; Ianita Zlateva, MPH; Alex Hutchinson, MBA; Jose Villagra, BS; and J. Nwando Olayiwola, MD, MPH
Electronic Health Record Problem Lists: Accurate Enough for Risk Adjustment?
Timothy J. Daskivich, MD, MSHPM; Garen Abedi, MD, MS; Sherrie H. Kaplan, PhD, MPH; Douglas Skarecky, BS; Thomas Ahlering, MD; Brennan Spiegel, MD, MSHS; Mark S. Litwin, MD, MPH; and Sheldon Greenfield, MD
Racial/Ethnic Variation in Devices Used to Access Patient Portals
Eva Chang, PhD, MPH; Katherine Blondon, MD, PhD; Courtney R. Lyles, PhD; Luesa Jordan, BA; and James D. Ralston, MD, MPH
Hospitalized Patients' and Family Members' Preferences for Real-Time, Transparent Access to Their Hospital Records
Michael J. Waxman, MD, MPH; Kurt Lozier, MBA; Lana Vasiljevic, MS; Kira Novakofski, PhD; James Desemone, MD; John O'Kane, RRT-NPS, MBA; Elizabeth M. Dufort, MD; David Wood, MBA; Ashar Ata, MBBS, PhD; Louis Filhour, PhD, RN; & Richard J. Blinkhorn Jr, MD

Measuring Overuse With Electronic Health Records Data

Thomas Isaac, MD, MBA, MPH; Meredith B. Rosenthal, PhD; Carrie H. Colla, PhD; Nancy E. Morden, MD, MPH; Alexander J. Mainor, JD, MPH; Zhonghe Li, MS; Kevin H. Nguyen, MS; Elizabeth A. Kinsella, BA; and Thomas D. Sequist, MD, MPH
Electronic health records data can accurately quantify overuse of clinical services and the risk factors that may trigger low-value testing and screening.
Our findings suggest that EHR data can be an important, although variable, source of information in identifying overuse of clinical services. For some measures of overuse, such as Pap smears in women younger than 21 years, structured EHR extracts were sufficient for identifying rates of overuse and relevant risk factors. In these cases, manual chart review added little insight into the potential clinical justification for a test or screening.

For most other measures, the combination of EHR data and manual chart review provided valuable information and elucidated some of the inherent complexities in overuse measures. For instance, cases of DEXAs in women younger than 65 years were easy to identify in the EHR, although clinical risk factors, such as previous osteopenia, were often identified in manual chart review. The initial diagnosis of osteopenia was often obtained from a DEXA that was not clearly indicated, thereby suggesting that an initial low-value test or screening may lead to subsequent low-value services. Further, there is debate on what the correct duration of follow-up should be after identifying mild or moderate osteopenia.25 Understanding these clinical nuances that may explain imaging, testing, or procedures is important, given the potential implication on costs (eg, the estimated cost of a single DEXA exam is approximately $125).26 The measure regarding overuse of antibiotics for sinusitis also proved challenging. Most cases of sinusitis identified using the EHR received antibiotics, which might represent a coding bias of clinicians in which the diagnosis is only listed when treated. 

Although there is variation in magnitude across measures, our study results suggest that EHR data provide important insights on overuse and presence of risk factors for several Choosing Wisely recommendations. However, both claims data and EHR data have limitations that can overestimate overuse, as the presence of risk factors is often only captured in chart review. We used manual chart review as the gold standard in our study, but recognize the labor-intensive nature of this methodology. Development of more automated text-based extracts (eg, natural language processing) could provide a less resource-intensive means to identify legitimate explanatory clinical risk factors of overuse. Further, as practices incorporate clinical decision support to identify low-value testing in real time and query providers to specify a clinical justification, the utility of EHR data extracts should improve. 


Our analysis has a number of limitations. We examined the use of EHR data in a large ambulatory care system that uses Epic software. The data warehouse structure and information available for procedures, medications, and laboratory tests likely have large variations compared with other EHRs. Second, we examined only a selection of Choosing Wisely recommendations, although the sample had variety in measures pertaining to medications, imaging, and procedures. Third, our study relied on manual chart review as the gold standard for determining overuse. Although manual chart review provides more clinical information than administrative claims data, we relied on information documented in the patient chart and therefore may be missing data that were not documented. Fourth, our chart reviews only examined clinical information from the encounter associated with the test order of each Choosing Wisely measure. A more thorough chart review looking back at previous notes and outside notes would likely yield more explanatory information, although this type of review requires more resources to perform. Finally, our EHR extracts and chart reviews examining explanatory factors and risk factors for each measure are open to clinical interpretation, and the clinical opinion of reviewers would impact the reproducibility of our results.


As clinicians and policy makers continue to gather data on overuse of low-value services, the methodologies and data sources utilized to measure overuse have become increasingly important. Developing more accurate and reliable calculations of overuse would be instrumental for policy makers and providers to identify opportunities for changing care delivery. Our work suggests that EHRs are an important source of data to quantify overuse and that EHRs can capture clinical information that often explains why a test or treatment is clinically indicated. Further, manual chart review, although more resource-intensive, may identify the presence of important risk factors that automated EHR data extracts cannot, and it should be considered alongside other methodologies of measuring overuse. The data from such manual chart reviews might be particularly important when engaging clinicians in the development and implementation of care delivery practices that reduce overuse of low-value services.

Author Affiliations: Atrius Health (TI), Newton, MA; Department of Health Policy and Management, Harvard T.H. Chan School of Public Health (MBR, ZL, KHN), Boston, MA; The Dartmouth Institute for Health Policy and Clinical Practice (CHC, NEM, AJM), Lebanon, NH; Norris Cotton Cancer Center, Dartmouth-Hitchcock Medical Center (CHC), Lebanon, NH; Department of Community and Family Medicine, Geisel School of Medicine at Dartmouth (NEM), Lebanon, NH; Division of General Internal Medicine, Brigham and Women’s Hospital (EAK, TDS), Boston, MA; Partners HealthCare (EAK, TDS), Boston, MA; Department of Health Care Policy, Harvard Medical School (TI, TDS), Boston, MA.

Source of Funding: The Commonwealth Fund.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

Authorship Information: Concept and design (TI, MBR, CHC, NEM, EAK, TDS); acquisition of data (TI, MBR, ZL, EAK, TDS); analysis and interpretation of data (TI, MBR, CHC, NEM, AJM, ZL, KHN, EAK, TDS); drafting of the manuscript (TI, MBR, CHC, NEM, AJM, KHN); critical revision of the manuscript for important intellectual content (TI, MBR, CHC, NEM, AJM, TDS); statistical analysis (MBR, CHC, ZL, KHN, TDS); obtaining funding (MBR, CHC, TDS); administrative, technical, or logistic support (CHC, AJM, KHN, EAK); and supervision (MBR). 

Address Correspondence to: Thomas Isaac, MD, Atrius Health, 275 Grove St, Ste 3-100, Auburndale, MA 02466. Email:

1. Schwartz AL, Chernew ME, Landon BE, McWilliams JM. Changes in low-value services in year 1 of the Medicare Pioneer Accountable Care Organization program. JAMA Intern Med. 2015;175(11):1815-1825. doi: 10.1001/jamainternmed.2015.4525.

2. About. Choosing Wisely website. Accessed December 11, 2015. 

3. Lists. Choosing Wisely website. Accessed December 11, 2015.

4. Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the Choosing Wisely campaign. JAMA Intern Med. 2015;175(12):1913-1920. doi: 10.1001/jamainternmed.2015.5441.

5. Grover M, McLemore R, Tilburt J. Clinicians report difficulty limiting low-value services in daily practice. J Prim Care Community Health. 2016;7(2):135-138. doi: 10.1177/2150131915624112.

6. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53-63.

7. Smith M, Saunders R, Stuckhardt L, McGinnis JM, eds. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: National Academies Press; 2013.

8. Colla CH. Swimming against the current—what might work to reduce low-value care? N Engl J Med. 2014;371(14):1280-1283. doi: 10.1056/NEJMp1404503.

9. Schwartz AL, Landon BE, Elshaug AG, Chernew ME, McWilliams JM. Measuring low-value care in Medicare. JAMA Intern Med. 2014;174(7):1067-1076. doi: 10.1001/jamainternmed.2014.1541.

10. Reid RO, Rabideau B, Sood N. Low-value health care services in a commercially insured population. JAMA Intern Med. 2016;176(10):1567-1571. doi: 10.1001/jamainternmed.2016.5031.

11. Charlesworth CJ, Meath TH, Schwartz AL, McConnell KJ. Comparison of low-value care in Medicaid vs commercially insured populations. JAMA Intern Med. 2016;176(7):998-1004. doi: 10.1001/jamainternmed.2016.2086.

12. Bhatia RS, Levinson W, Shortt S, et al. Measuring the effect of Choosing Wisely: an integrated framework to assess campaign impact on low-value care. BMJ Qual Saf. 2015;24(8):523-531. doi: 10.1136/bmjqs-2015-004070.

13. Hong AS, Ross-Degnan D, Zhang F, Wharam JF. Small decline in low-value back imaging associated with the ‘Choosing Wisely’ campaign, 2012-14. Health Aff (Millwood). 2017;36(4):671-679. doi: 10.1377/hlthaff.2016.1263.

14. Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc. 2007;14(1):10-15. doi: 10.1197/jamia.M2198.

15. Elshaug AG, McWilliams JM, Landon BE. The value of low-value lists. JAMA. 2013;309(8):775-776. doi: 10.1001/jama.2013.828.

16. Bailey SR, Heintzman JD, Marino M, et al. Measuring preventive care delivery: comparing rates across three data sources. Am J Prev Med. 2016;51(5):752-761. doi: 10.1016/j.amepre.2016.07.004.

17. Naessens JM, Ruud KL, Tulledge-Scheitel SM, Stroebel RJ, Cabanela RL. Comparison of provider claims data versus medical records review for assessing provision of adult preventive services. J Ambul Care Manage. 2008;31(2):178-186. doi: 10.1097/01.JAC.0000314708.65289.3b.

18. Baker DW, Qaseem A, Reynolds PP, Gardner LA, Schneider EC; American College of Physicians Performance Measurement Committee. Design and use of performance measures to decrease low-value services and achieve cost-conscious care. Ann Intern Med. 2013;158(1):55-59. doi: 10.7326/0003-4819-158-1-201301010-00560.

19. Heintzman J, Bailey SR, Hoopes MJ, et al. Agreement of Medicaid claims and electronic health records for assessing preventive care quality among adults. J Am Med Inform Assoc. 2014;21(4):720-724. doi: 10.1136/amiajnl-2013-002333.

20. Kern LM, Malhotra S, Barrón Y, et al. Accuracy of electronically reported “meaningful use” clinical quality measures: a cross-sectional study. Ann Intern Med. 2013;158(2):77-83. doi: 10.7326/0003-4819-158-2-201301150-00001.

21. Schlemmer E, Mitchiner JC, Brown M, Wasilevich E. Imaging during low back pain ED visits: a claims-based descriptive analysis. Am J Emerg Med. 2015;33(3):414-418. doi: 10.1016/j.ajem.2014.12.060.

22. Colla CH, Morden NE, Sequist TD, Schpero WL, Rosenthal MB. Choosing wisely: prevalence and correlates of low-value health care services in the United States [erratum in J Gen Intern Med. 2016;31(4):450. doi: 10.1007/s11606-015-3420-5]. J Gen Intern Med. 2015;30(2):221-228. doi: 10.1007/s11606-014-3070-z.

23. Colla CH, Sequist TD, Rosenthal MB, Schpero WL, Gottlieb DJ, Morden NE. Use of non-indicated cardiac testing in low-risk patients: Choosing Wisely. BMJ Qual Saf. 2015;24(2):149-153. doi: 10.1136/bmjqs-2014-003087.

24. Backhus LM, Farjah F, Varghese TK, et al. Appropriateness of imaging for lung cancer staging in a national cohort. J Clin Oncol. 2014;32(30):3428-3435. doi: 10.1200/JCO.2014.55.6589.

25. Gourlay ML, Fine JP, Preisser JS, et al; Study of Osteoporotic Fractures Research Group. Bone-density testing interval and transition to osteoporosis in older women. N Engl J Med. 2012;366(3):225-233. doi: 10.1056/NEJMoa1107142.

26. Bone-density tests. Choosing Wisely website. Published May 2012. Accessed December 11, 2015.
Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up