Currently Viewing:
The American Journal of Managed Care December 2014
Quality of End-of-Life Care for Cancer Patients: Does Home Hospice Care Matter?
Netta Bentur, PhD; Shirli Resnizky, MA; Ran Balicer, MD; and Tsofia Eilat-Tsanani, MD
Out-of-Plan Pharmacy Use: Insights Into Patient Behavior
Thomas Delate, PhD; Alexander P. Block, PharmD; Deanna Kurz, BA; and Sarah J. Billups, PharmD
Paying for Telemedicine
Robert S. Rudin, PhD; David Auerbach, PhD; Mikhail Zaydman, BS; and Ateev Mehrotra, MD
Currently Reading
Validating Electronic Cancer Quality Measures at Veterans Health Administration
Jeremy B. Shelton, MD, MSHS; Ted A. Skolarus, MD, MPH; Diana Ordin, MD, MPH; Jennifer Malin, MD, PhD; AnnaLiza Antonio, MS; Joan Ryoo, MD, MSHS; and Christopher S. Saigal, MD
ACO Contracting With Private and Public Payers: A Baseline Comparative Analysis
Valerie A. Lewis, PhD; Carrie H. Colla, PhD; William L. Schpero, MPH; Stephen M. Shortell, PhD, MPH, MBA; and Elliott S. Fisher, MD, MPH
Reference-Based Pricing: An Evidence-Based Solution for Lab Services Shopping
L. Doug Melton, PhD, MPH; Kent Bradley, MD, MPH, MBA; Patricia Lin Fu, MPH; Raegan Armata, BS, MBA; and James B. Parr, BA
Addressing Cost Barriers to Medications: A Survey of Patients Requesting Financial Assistance
David Grande, MD, MPA; Margaret Lowenstein, MD, MPhil; Madeleine Tardif, BA; and Carolyn Cannuscio, ScD
Preconsultation Exchange in the United States: Use, Awareness, and Attitudes
Justin L. Sewell, MD, MPH; Katherine S. Telischak, MSc; Lukejohn W. Day, MD; Neil Kirschner, PhD; and Arlene Weissman, PhD
Medicare Star Excludes Diabetes Patients With Poor CVD Risk Factor Control
Julie Schmittdiel, PhD; Marsha Raebel, PharmD; Wendy Dyer, MS; John Steiner, MD, MPH; Glenn Goodrich, MS; Andy Karter, PhD; and Gregory Nichols, PhD
There's More Than One Way to Build a Medical Home
Manasi A. Tirodkar, PhD, MS; Suzanne Morton, MPH, MBA; Thomas Whiting, MPA; Patrick Monahan, MD; Elexis McBee, DO; Robert Saunders, PhD; and Sarah Hudson Scholle, DrPH, MPH
Improving Medication Understanding Among Latinos Through Illustrated Medication Lists
Arun Mohan, MD, MBA; M. Brian Riley, MA; Brian Schmotzer, MS; Dane R. Boyington, PhD; and Sunil Kripalani, MD, MSc
Predicting Nursing Home Placement Among Home- and Community-Based Services Program Participants
Melissa A. Greiner, MS; Laura G. Qualls, MS; Isao Iwata, MD, PhD, EdM; Heidi K. White, MD; Sheila L. Molony, PhD, APRN, GNP-BC; M. Terry Sullivan, RN, MSW, MSN; Bonnie Burke, MS; Kevin A. Schulman, MD; and Soko Setoguchi, MD, DrPH

Validating Electronic Cancer Quality Measures at Veterans Health Administration

Jeremy B. Shelton, MD, MSHS; Ted A. Skolarus, MD, MPH; Diana Ordin, MD, MPH; Jennifer Malin, MD, PhD; AnnaLiza Antonio, MS; Joan Ryoo, MD, MSHS; and Christopher S. Saigal, MD
Even in a fully integrated healthcare system, only 28% of cancer quality measures could be validated by using electronically available data.
To assess the feasibility and validity of developing electronic clinical quality measures (eCQMs) of cancer care quality from existing metrics, using electronic health records, administrative, and cancer registry data.

Study Design
Retrospective comparison of quality indicators using chart abstracted versus electronically available data from multiple sources.

We compared the sensitivity and specificity of eCQMs created from structured data from electronic health records (EHRs) linked to administrative and cancer registry data to data abstracted from patients’ electronic health records. Twenty-nine measures of care were assessed in 15,394 patients with either incident lung or prostate cancer from 2007 and 2008, respectively, and who were treated in the Veteran’s Health Administration (VHA).

It was feasible to develop eCQMs for 11 of 18 (61%) lung cancer measures, 4 (22%) of which were considered to be valid measures of the care constructs. Among prostate cancer measures, 6 of 11 (55%) were feasible, and 4 (36%) were both feasible and valid. Of the 29 metrics, data was available to create eCQMs for 17 (59%) cancer care metrics, and 8 (28%) were considered valid.

In a large integrated healthcare system with nationally standardized electronic health records, administrative, and cancer registry data, 28% of cancer quality measures developed for chart abstraction could be translated into valid eCQMs. These results raise much concern about the development of electronic clinical quality measures for cancer care, particularly in healthcare environments where data are disparate in both form and location.

Am J Manag Care. 2014;20(12):1041-1047
  • Only 28% of cancer quality measures could be translated into electronic clinical quality metrics, even in the largest integrated healthcare system in the United States with nationally standardized electronic data repositories.
  • These results raise concern about the use of electronic clinical quality measures for cancer care, particularly outside integrated healthcare environments, and suggest that it may be premature to think that meaningful measures of cancer care delivery can be developed for widespread use across electronic health record platforms.
  • Current health data systems are inadequate to support meaningful electronic quality measurement in cancer care.
As the healthcare system moves toward a paradigm that rewards “value” in healthcare, or the maximizing of quality while minimizing costs, accurate measurement of the quality of healthcare delivery has taken center stage in reform efforts.1-5 Currently, most quality measures rely on for data acquisition chart abstraction, which is considered the gold standard. While chart abstraction allows access to all types of medical data, clinical notes in particular, it is time-consuming and prohibitively expensive to permit broad and timely measurement of care quality.6 It has long been hoped that electronic measurement of care delivery would open the door to timely, meaningful, and actionable information for many stakeholders.7-10 Electronic health records (EHRs), now adopted by more than 70% of physicians, and health information exchanges are 2 key components of the visions of data interoperability.7,11,12 This is coupled with technical developments, like natural language processing (NLP) and structured clinical templates, which are expected to facilitate computerized access to the content of free text that underlies much of primary clinical documentation, and13,14 payment reforms, like accountable care organizations, which encourage linkage of healthcare data silos.13-17 While human chart abstractors are able to bridge data silos and interpret the full range of available data sources, it is still unclear if the above reforms will make electronic clinical quality measures (eCQMs) and measurement feasible. Many studies have illustrated wide gaps in agreement among electronically available data sources and between these sources and chart abstraction.18-27 Furthermore, eCQMs that are solely based on administrative data have been criticized for failing to measure what matters.28 Cancer care, additionally, is especially challenging to measure, in part because it relies on the multifactorial process of staging and because care is typically delivered across multiple specialties and care settings.

Large integrated healthcare systems, like the Veterans Health Administration (VHA), having already achieved national data integration across all aspects of care delivery, with a common EHR and national, standardized, electronically available data warehouses, offer an opportunity to examine the feasibility and validity of using available electronic data to measure cancer care quality.29-32 Recently, the VHA performed quality assessments of care for lung and prostate cancer patients using manual chart abstraction.33 We hypothesized that the data needed to measure quality performance was available in the VHA’s electronic clinical, administrative, and cancer registry repositories and could be used to create eCQMs from existing quality measures.


Study Design

This was a retrospective, national VHA cohort study sponsored by the VHA Office of Analytics and Business Intelligence. The VHA Greater Los Angeles Institutional Review Board approved this study.

Quality Measures

VHA expert panels reviewed the relevant literature and proposed quality measures—using the RAND/ UCLA modified Delphi technique, for national assessments of the quality of care for lung cancer and prostate cancer diagnosed in 2007 and 2008, respectively—to be performed using data abstracted from patients’ EHRs.34 There were 29 quality metrics (18 for lung cancer and 11 for prostate cancer) spanning the continuum of care (see Appendix for more detailed descriptions). Many measures replicated, or were similar to, metrics endorsed by non-VHA organizations (eAppendix). Twenty measures addressed diagnosis, treatment, and management; 5 addressed supportive care; and 4 addressed end-of-life care. Twenty-six measures were patient level, 2 were visit level, and 1 was medical center level.

Data Sources

Data necessary for 29 measures of the quality of lung and prostate cancer care were abstracted from patients’ VHA Computerized Patient Record System (CPRS) medical records by External Peer Review Program (EPRP) nurse abstractors conducting the national VHA assessments of the quality of cancer care (chart abstraction cohort). Using an abstraction tool specifically developed for the quality assessments, EPRP abstractors systematically abstracted the data necessary to score the quality indicators. Case-level quality indicator and timeliness results were provided to VAMCs for review and correction, and the final results were calculated based on these field-corrected data.

Data for 29 eCQMs were obtained from extracts from the VHA’s EHR through the Corporate Data Warehouse (CDW) and Decision Support System (DSS), both of which are national, standardized, and near real-time clinical and administrative data repositories; and from the VHA Central Cancer Registry (VACCR). The eCQM data set was created by linking data from the VACCR, CDW and DSS. This included inpatient and outpatient encounter and procedure coding, as well as pharmacy, lab, radiology, allergy, problem list, and vital sign data.

Study Population

In both the chart abstraction and eCQM cohorts, case identification criteria were identical: all newly diagnosed, pathologically confirmed cases of lung cancer during 2007 (n = 8125) and prostate cancer during 2008 (n = 12,572) that were reported to the VACCR. Patients were excluded if the pathologic diagnosis was not identified in the EHR at the index facility that reported the case to the registry (lung, 1297; prostate, 489), a situation that could occur if patients were treated at more than 1 facility.

Patients were excluded from the chart abstraction cohorts if they were diagnosed at autopsy; death occurred in 30 days or less after cancer diagnosis; hospice enrollment occurred 30 days or less after cancer diagnosis (lung, 947; prostate, 16); they were enrolled if they had a pre-existing or concurrent diagnosis of metastatic cancer other than lung or prostate cancer (lung, 540; prostate, 627); they were enrolled in a cancer clinical trial (lung, 57; prostate, 147); there was documentation of comfort measures only in a hospital discharge summary or nursing home note 30 days or less after cancer diagnosis (lung, 91; prostate, 14); or documentation of life expectancy of 6 months or less in their Problem List at the time of diagnosis (lung, 39; prostate, 4). The resulting lung cancer and prostate cancer abstraction cohorts included 4865 and 11,211 patients, respectively.

Patients were excluded from the lung and prostate cancer eCQM cohorts if they died within 60 days of diagnosis (lung, 1482; prostate, 674); if there was no pathologic confirmation of disease according to the VACCR (lung, 419; prostate, 49); or if there were less than 2 encounters with an International Classificiation of Diseases, Ninth Revision (ICD-9) diagnosis code for prostate or lung cancer (lung, 229; prostate, 8). The eCQM cohort included 5995 patients with lung cancer and 11,700 patients with prostate cancer.

The final validation cohort included patients common to both the lung and prostate abstraction and the eCQM cohorts, n = 4865 and n = 10,529, respectively.

Adaptation of quality measures for eCQMs. Using the linked eCQM data set, we identified the elements specified by each quality indicator and, where more than 1 option existed, determined the optimal specification iteratively. Two measures, “No adjuvant chemotherapy for stage IA NSCLC” and “No radiation therapy for resected stage I, II NSCLC,” were excluded from analysis prior to eCQM creation because adherence, by chart abstraction, was greater than 99% with no variation seen across facilities.

Data Analysis

eCQMs were considered feasible if electronically available data could specify each aspect of the measure. Validity was evaluated by comparing the sensitivity and specificity of the denominator and numerator for each measure, using chart abstraction as the gold standard, and by comparing overall pass rates between the electronic and the chart-abstracted versions of the measures. Although there is no accepted cut-point for numerator and denominator sensitivity and specificity in determining measure validity, nor an accepted level of agreement between pass rates that marks validity, we considered at least 80% specificity for both the denominator and the numerator (compared with chart abstraction data) as valid for an eCQM. However, quality measures with less than 80% specificity or sensitivity were assessed individually and considered likely valid as eCQMs if internal validity could be supported—even if results from measure use might not be comparable across data collection methodologies. Analyses were performed using SAS statistical software, version 9.1 (Cary, North Carolina).


Patient Characteristics

Patient demographic characteristics were similar to national trends in lung and prostate cancer (Table 1). The distribution of patients by AJCC stage (lung) or D’Amico risk group classification (prostate) were similar in both the chart abstraction cohort and the VACCR, with 25.5%, 7.8%, 28%, and 34.2% of lung cancer patients with Stages I, II, III, and IV, respectively.35,36 Among men with prostate cancer in the chart abstraction cohort, 24.5%, 34.8%, and 26.3% had low-, intermediate-, and high-risk disease, respectively. The overall percent agreement in staging classification between the VACCR and chart abstraction was 65% (95% CI, 63-67) among lung cancer cases and 75% (95% CI, 74-77) for D’Amico risk classification in prostate cancer cases.

Feasibility of eCQM creation. Using the VHA EHR, administrative, and VACCR data, it was possible to specify the necessary data elements for 17 of 29 (59%) of the eCQMs (Table 2, eAppendix). This included 6 of the 11 prostate cancer quality measures (55%) and 11 of 18 lung cancer quality measures (61%). In some cases, to accommodate available electronic data while attempting to preserve the intent and face validity of the measure, changes were required of the quality measure definitions. For example, the numerator of “surgical node sampling” in lung cancer treatment was modified to include cases based on the number of lymph nodes sampled instead of the number of lymph node stations that were sampled. While the number of nodes has value in determining staging, and may correlate with the number of lymph node stations, it is not an exact match and limits the ability to compare quality scores using the 2 different approaches.37 Measures for which it was not feasible to create eCQMs are listed in Table 3.

Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up