Currently Viewing:
The American Journal of Managed Care June 2012
Thromboembolism Prophylaxis in Medical Inpatients: Effect on Outcomes and Costs
Onur Baser, MS, PhD; Nishan Sengupta, PhD; Anne Dysinger, MA; and Li Wang, MA, PhD
Racial Disparities in Lipid Control in Patients With Diabetes
Darcy Saffar, MPH; L. Keoki Williams, MD, MPH; Jennifer Elston Lafata, PhD; George Divine, PhD; and Manel Pladevall, MD, MS
Currently Reading
Automating Care Quality Measurement With Health Information Technology
Brian Hazlehurst, PhD; Mary Ann McBurnie, PhD; Richard A. Mularski, MD, MSHS, MCR; Jon E. Puro, MPA-HA; and Susan L. Chauvie, RN, MPA-HA
Bowling Alone, Healing Together: The Role of Social Capital in Delivery Reform
Sachin H. Jain, MD, MBA; Riya Goyal, BA; Susannah Fox, BA; and William H. Shrank, MD, MS, HS
Quality Care Opportunities: Refining Physician Performance Measurement in Ambulatory Care
Kimberly M. Lovett, MD; and Bryan A. Liang, MD, PhD, JD
Unintended Consequences of a Quality Measure for Acute Bronchitis
Serena Roth, MD; Ralph Gonzales, MD, MSPH; Tammy Harding-Anderer, PhD; Frederick J. Bloom, Jr, MD, MMM; Thomas Graf, MD; Melissa S. Stahl, MPH; Judith H. Maselli, MSPH; and Joshua P. Metlay, MD, PhD
Impact of Positive Airway Pressure Among Obstructive Sleep Apnea Patients
Qian Cai, MS, MSPH; Hiangkiat Tan, MS, BPharm; and Joseph Singer, MD
Barriers to Transition of Obese Patients From Hospital to Community
Jane Miles, MSN, RN, NEA-BC; Darren P. Anderson, MSN, RN, ACM; Martha Engelke, PhD; Mary K. Kirkpatrick, RN, MSN, EdD; Mary Lisa Pories, MSW; Wanda G. Waters, RN, BSN; Frank R. Watkins, RN, CCRC; Marie E. Pokorny, RN, MSN, PhD; and Mary Ann Rose, MSN, EdD

Automating Care Quality Measurement With Health Information Technology

Brian Hazlehurst, PhD; Mary Ann McBurnie, PhD; Richard A. Mularski, MD, MSHS, MCR; Jon E. Puro, MPA-HA; and Susan L. Chauvie, RN, MPA-HA
The authors discuss the design and evaluation of a health information technology platform that enables comprehensive, automated assessment of care quality in electronic medical records.
To assess how well our implementation of the ACQ measure set performed, we carried out a validation process using chart review for 818 patients randomly selected from among those identified to have persistent asthma by our method, stratified by age and sex for each health system (443 at KPNW and 375 at OCHIN). Each stratum was populated with 3 to 10 distinct patients who had an exacerbation within the chart review time period. This allowed us to compare the overall accuracy, sensitivity, and specificity of the ACQ measures by site, relative to results obtained by manual chart abstraction performed by trained abstractors (the reference standard).

Most ACQ measures performed relatively well in the KPNW system (Table 2). Measure accuracy (agreement with chart review) ranged from 63% to 100% and averaged 88% across all measures (95% confidence interval [CI] 82%-93%). Mean sensitivity was 77% (95% CI 62%-92%) and was 60% or greater for 15 of the 18 measures (and 90% or greater for 9 of those measures). Similarly, mean specificity was 84% (95% CI 75%-93%), with 15 measures having specificity of 60% or greater (9 measures with specificity of 90% or greater). For 2 measures, specificity was more than 90% but sensitivity was poor. The measure attempting to ascertain whether a history or review of prior hospitalizations and emergency department visits had been obtained failed to identify any of the 5 patients noted by abstractors to have received this care. In addition, documentation of patient education in the case of newly prescribed inhaled therapy had sensitivity of just 12%, identifying only 3 of 25 patients noted by the abstractors to have received this care. There was only 1 patient on theophylline in the KPNW chart review sample, precluding estimation of the accuracy of this measure.

The automated ACQ analysis was less accurate against the OCHIN system (Table 2). These differences came not from the level of quality found in the 2 systems but from differences in documentation, EMR implementation, and clinical practice that our method is not yet properly accommodating. Mean overall accuracy was 80% (95% CI 72%-89%) and ranged from 36% to 99% across all measures. Mean sensitivity and specificity were 52% (95% CI 35%-69%) and 82% (95% CI 69%-95%), respectively. Performance was better among the routine measures compared with the exacerbation-related measures. Among the 11 routine care measures, 8 had specificities higher than 80% and 5 had sensitivities higher than 80%. Of these measures, 3 had specificities of 50% or lower, while another 5 measures had sensitivities of 50% or lower. Of the 7 exacerbation-related measures, 5 were evaluable at OCHIN (assessment was not possible for 2 of the exacerbation measures: no patients on theophylline were identified, and since hospital discharge information was unavailable, the 4-week follow-up contact prescribed by measure 16 was not evaluable). Among the 5 evaluable measures for exacerbation care, overall accuracy ranged from 36% to 96%. Sensitivity tended to be low (5.3% to 58.1%), while specificities were generally high (95% or higher for 4 of the 5 measures).


Across the evaluable measures at each site, specificity was similar: 9 of 16 measures reached 90% or better. The largest difference between sites was seen in measure sensitivity. While most measures in KPNW reached 60% sensitivity (15 of 18 measures), only a minority (6 of 16 measures) met or exceeded 60% sensitivity in OCHIN. Of the 9 measures with sensitivity below 50% at OCHIN, 6 of them rely exclusively on text processing of clinical notes, indicating that our textprocessing solution needs to be further refined to identify the relevant clinical events the abstractors observed in the OCHIN EMR. Potential explanations for this discrepancy in performance of the automated measures between our 2 study sites include (1) the possibility that the chart reviewer was observing events in comment fields of the OCHIN records that were not available to the automated program and (2) the possibility that there may be greater variability in how and where OCHIN providers document visits. Each of these reasons could explain the inability of our automated method to identify the necessary events in the OCHN medical records. Additional modification of the specifications of the automated method will be needed to capture these differences across medical records. Another likely explanation is that the OCHIN EMR is quite new compared with KPNW’s, having been implemented in 2005. Hence, familiarity with the EMR and documentation support resources may have affected completeness and consistency of clinical data entry in some cases.

The automated method described here utilizes the clinically rich, heterogeneous data captured in transactional EMR systems to assess the quality of care delivered to patients with persistent asthma. One question that arises is whether administrative data alone could suffice to perform this task. Leaving aside the fidelity issues inherent in using billing data to compute clinical care process metrics, we found that at most 6 of the 22 ACQ measures (and 5 of the 18 measures we implemented) could be addressed partially or in whole with administrative data alone. In short, the use of administrative data alone would not meet the goals of assessing compliance with current guidelines for best practices in caring for patients with persistent asthma.


Comprehensive and routine quality assessment requires both state-of-the-art EMR implementation and an adaptable health information technology platform that enables automated measurement of complex clinical practices. We designed a system to respond to these challenges and implemented it in 2 diverse healthcare systems to assess outpatient asthma care. These automated measures generally performed well in the HMO setting, where clinical practice is more standardized; additional refinement is needed for health systems that encompass more diversity in clinical practice, patient population, and setting.

Our design overcomes many challenges created by textbased guidelines, nonstandard data elements, and text clinical notes. Although we have only been able to implement 18 of the 22 measures to date, and although chart review showed that some may require refinement, the automated approach promises to be more affordable than chart review. Future work will explore whether our design will accommodate all of the ACQ measures and whether our implementation can be enhanced to improve performance in more diverse healthcare systems.

Author Affiliations: From Center for Health Research (BH, MAM, RAM), Kaiser Permanente Northwest, Portland, OR; OCHIN Inc (JEP, SLC), Portland, OR.

Author Disclosures: Dr Hazlehurst reports employment with Kaiser Permanente and has received and has pending grants from the Agency for Healthcare Research and Quality, the funder of this study. Mr Puro reports receiving grants from Obesity Care Quality and the Comparative Effectiveness Research Hub. Ms Chauvie reports receiving grants from Obesity Care Quality. The other authors (MAM, RAM) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (BH, RAM, JEP, SLC); acquisition of data (BH, JEP, SLC); analysis and interpretation of data (BH, MAM, RAM, JEP); drafting of the manuscript (BH, MAM, RAM, SLC); critical revision of the manuscript for important intellectual content (BH, MAM, RAM, JEP, SLC); statistical analysis (MAM, RAM, JEP); provision of study materials or patients (BH); obtaining funding (BH, RAM); administrative, technical, or logistic support (BH, RAM, JEP, SLC); and supervision (BH, JEP).

Funding Source: Agency for Healthcare Research and Quality, grant R18-HS17022.

Address correspondence to: Brian Hazlehurst, PhD, Center for Health Research, 3800 N Interstate Ave, Portland, OR 97227-1110. E-mail:
1. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635-2645.

2. Corrigan J, Donaldson MS, Kohn LT, eds. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.

3. Hazlehurst B, McBurnie M, Mularski R, Puro J, Chauvie S. Automating quality measurement: a system for scalable, comprehensive, and routine care quality assessment. AMIA Annu Symp Proc. 2009:229-233.

4. Mularski RA, Asch SM, Shrank WH, et al. The quality of obstructive lung disease care for adults in the United States as measured by adherence to recommended processes. Chest. 2006;130(6):1844-1850.

5. Kerr EA, Asch SM, Hamilton EG, McGlynn EA. Quality of Care for Cardiopulmonary Conditions: A Review of the Literature and Quality Indicators. Santa Monica, CA: RAND; 2000.

6. National Asthma Education and Prevention Program, Third Expert Panel on the Diagnosis and Management of Asthma. Expert Panel Report 3: Guidelines for the Diagnosis and Management of Asthma. Bethesda, MD: National Heart, Lung and Blood Institute; August 2007. Accessed April 17, 2012.

7. Dolin RH, Alschuler L, Boyer S, et al. HL7 Clinical Document Architecture, Release 2. J Am Med Inform Assoc. 2006;13(1):30-39.

8. Hazlehurst B, Frost HR, Sittig DF, Stevens VJ. MediClass: a system for detecting and classifying encounter-based clinical events in any electronic medical record. J Am Med Inform Assoc. 2005;12(5):517-529.

9. Hazlehurst B, Sittig DF, Stevens VJ, et al. Natural language processing in the electronic medical record: assessing clinician adherence to tobacco treatment guidelines. Am J Prev Med. 2005;29(5):434-439.

10. Hazlehurst B, Naleway A, Mullooly J. Detecting possible vaccine adverse events in clinical notes of the electronic medical record. Vaccine. 2009;27(14):2077-2083.
Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up