Jennifer Malin, MD, PhD

To determine whether a treating oncologist’s characteristics are associated with variation in use of chemotherapy for patients with advanced non—small cell lung cancer (aNSCLC) at the end of life.

Using the 2009 Surveillance, Epidemiology, and End Results—Medicare database, we studied chemotherapy receipt within 30 days of death among Medicare enrollees who were diagnosed with aNSCLC between 1999 and 2006, received chemotherapy, and died within 3 years of diagnosis. A multilevel model was constructed to assess the contribution of patient and physician characteristics and geography to receiving chemotherapy within 30 days of death.

 Among 21,894 patients meeting eligibility criteria, 43.1% received chemotherapy within 30 days of death. In unadjusted bivariate analyses, female sex, Asian or black race, older age, and a greater number of comorbid diagnoses predicted lower likelihood of receiving chemotherapy at the end of life (

 ≤.038 for all comparisons). Adjusting for patient and physician characteristics, physicians in small independent practices were substantially more likely than those employed in other practice models, particularly academic practices or nongovernment hospitals, to order chemotherapy for a patient in the last 30 days of life (

 <.001 for all comparisons); female physicians were less likely than males to prescribe such treatment (

Patients receiving care for aNSCLC in small independent oncology practices are more likely to receive chemotherapy in the last 30 days of life.

Oncologists’ characteristics explain significant variation in patients’ receipt of chemotherapy in the last 30 days of life:

		Patients should understand the variation in practices among oncologists treating the same condition.

		Awareness of such variation may influence an individual oncologist’s practice decisions and eventually lead to consensus practices at end of life; practices may already have changed since the period under study.

		Less variation is likely to yield better alignment between patient goals and treatment received, and result in higher value care at the end of life.

		Payers may wish to consider oncologist practice type in determining network participation.

Despite a half century of treatment advances, lung cancer—the most common solid tumor in the United States&mdash;remains among the cancers least responsive to treatment.

 A majority of patients are diagnosed at an advanced stage and, even with the newest therapies, barely 1 in 10 are alive a year after diagnosis.

 In the final weeks of life, many patients with advanced non—small cell lung cancer (aNSCLC) undergo aggressive treatment that can include repeated emergency department visits, prolonged hospitalizations, intensive care, and additional lines of chemotherapy that contravene guidelines.

 Attendant effects on quality of life are well documented, and the value of such expensive treatment is debated in the lay, clinical, and policy realms.

Why aggressive end-of-life treatment occurs is not clearly understood. Regional variations in the aggressiveness of cancer treatment have been well established and patient factors have been explored

; however, the degree of variation between individual physicians is not known.

 We conducted this study to determine the extent to which physician characteristics explain patients’ receipt of chemotherapy in the 30 days prior to death among patients with aNSCLC.

We used the 2009 Surveillance, Epidemiology, and End Results (SEER) cancer registry and linked Medicare claims

 to identify patients and describe treatment patterns. SEER regions include 28% of the US population. Approximately 68% of lung cancers diagnosed in SEER regions are in Americans older than 65 years.

 Using the Unique Physician Identification Number Registry, we linked the physician submitting a Medicare claim to their characteristics in the American Medical Association (AMA) Masterfile.

 details the initial sample and number of patients dropped due to each exclusion criterion. We first identified all patients in the SEER registry diagnosed with lung cancer between 1999 and 2006. We selected 1999 due to changes in physicians’ unique identification codes that year and 2006 to allow for up to 3 years of data after diagnosis. We limited the sample to those 65 years or older at diagnosis who were enrolled in traditional Medicare Part A and Part B from at least 12 months prior to a first diagnosis of lung cancer until 3 years after diagnosis or until death. Patients with other incident cancers were excluded to avoid erroneously counting chemotherapy directed toward another cancer.

Among 193,200 subjects satisfying all conditions (ie, lung cancer, age ≥65, continuous enrollment in Medicare A and B, no other cancer), we excluded 4518 subjects who died on unknown dates and 123 subjects with charges for chemotherapy after their recorded date of death. Our cohort was then limited to 155,794 patients with stage 3b or 4 aNSCLC; an additional 38,311 were excluded for diagnosis dates out of range. We excluded 89,069 who had not received chemotherapy within 3 years of initial diagnosis and compared their characteristics with those who received chemotherapy. To avoid insufficient data biasing interpretation of physicians’ practice patterns, subjects were excluded if not treated by an identifiable oncologist who provided care to 5 or more patients in the sample. In sensitivity analyses, we evaluated the impact of physician characteristics on receipt of chemotherapy in the last 30 days of life for all physicians, irrespective of the number of patients seen, and physicians with 10 or more patients. Our final analytic sample comprised 21,894 aNSCLC subjects.

Chemotherapy use was established by Medicare charges in outpatient, inpatient, or physician claims for chemotherapy-related encounters (

International Classification of Diseases, Ninth Revision, Clinical Modification

] diagnosis codes V58.1, V66.2, and V67.2), chemotherapy administration (

 procedure code 99.25; Current Procedural Terminology codes 96400-96549; Health Care Common Procedure Coding System codes J8530, J8560, J8600, J8610, J8999, Q0083-Q0085, and G0921-G0932; and Revenue Center Codes 0331, 0332, and 0335), or chemotherapy agents (J Codes other than diethylstilbestrol and leuprolide). The primary outcome was whether a patient who ever received chemotherapy received a final dose during the last 30 days of life.

Administration of chemotherapy at the end of life was attributed to the oncologist submitting a Medicare claim with the latest date of service rendered prior to a patient’s death. Sensitivity analysis was performed, attributing the patient to the oncologist with the most visits. Physicians were considered oncologists if the billing physician’s specialties in either Medicare claims or AMA-linked physician files included oncology, hematology-oncology, or hematology.

Patient were classified by race (white, black, Hispanic, Asian, other), sex, age at diagnosis (65-69, 70-74, 75-79, 80-84, ≥85 years), last known marital status, median income by zip code (by quartile, as a proxy for socioeconomic status), and year of diagnosis. Time between diagnosis and last chemotherapy was calculated and grouped (<1, 1, 2-3, 4-5, 6-7, 8-9, 10-11, 12-23, 24-36 months), as it was hypothesized that a recent diagnosis might be associated with receiving treatment at the end of life. We also included the proportion of blacks in the patient’s residential zip code and birthplace outside of the United States.

We calculated a modified Charlson Comorbidity Index score for each patient using 

—coded diagnoses from inpatient claims, carrier claims, and outpatient claims using the Wang method.

 In order to approximate patients’ health leading up to death, comorbidity scores were calculated from claims for services provided during the 12 months prior to the month of death.

Physician characteristics included in the model were sex and year that medical training was completed. Age was strongly correlated with year of training completion and thus excluded. We examined the type of practice based on the present employer variable from the AMA Masterfile and classified this variable into 6 categories: small independent (physician-owned, 1-2 physicians), group practice (physician-owned, >2 physicians), government (employed by city, county, state, or federal government), academic (employed by medical schools), hospital (employed by non—government-owned hospital), and other.

As there is good evidence supporting geographic variations in treatment practices, we sought to control for such variation based on SEER registry sites; however, because of its size and previously demonstrated practice variation,

 California was split by county into 4 zones: Los Angeles, non-LA Metro-South (San Diego, Riverside, San Bernardino, Orange, Ventura), Metro-North (San Francisco, Alameda, Santa Clara, Contra Costa, San Mateo, Marin), and Other. Rural Georgia, with just 47 eligible cases, was combined with Atlanta, yielding a single site for all of Georgia. SEER sites were otherwise categorized according to the SEER 17 registry.

Frequency distributions were calculated for patient, oncologist, and geographic variables. We used a multilevel logistic regression mixed model with dichotomous outcomes to estimate the probability of receiving chemotherapy treatment in the last 30 days of life. Patients were nested within physicians, who, in turn, were nested within geographic locations (SEER site, modified as above) as a random intercept at the highest level. The model adjusted for the patient and physician covariates, as described above. To facilitate interpretation of the magnitude of the effects, adjusted relative risks are presented along with the coefficient estimates and 

To calculate the marginal effect of the physician’s type of practice on receiving chemotherapy at the end of life, each patient’s probability of receiving treatment was recalculated as if all received treatment under a uniform type of practice, adjusting for patient variables and other physician variables. This was repeated for each type of practice.

University of California, Los Angeles, Institutional Review Board approved the study.

We identified 23,687 continuously enrolled Medicare (parts A and B) patients diagnosed with aNSCLC between 1999 and 2006. 

 shows the distribution of patient characteristics and associated probability of receiving chemotherapy within 30 days of death, among the 21,894 (92.4%) patients receiving chemotherapy within 3 years of diagnosis. Of these, 9447 (43.1%) received chemotherapy within 30 days of death.

In bivariate analyses, men were more likely than women to receive chemotherapy near the end of their lives (45.9% vs 39.5%; 

 <.001) (Table 1). Patients were less likely to receive chemotherapy at the end of life if they were Asian (33.3%) or black (40.0%) compared with whites (43.9%; 

 <.001), older (40.8% among those aged ≥85 vs 44.4% among those aged 65-69; 

 = .001), or diagnosed in 2005 (37.8%) or 2006 (37.5%) compared with 1999 (43.9%) (

 <.001). Despite these differences in treatment, none of these variables were associated with a difference in survival (data not shown).

Characteristics of the 89,069 patients excluded for nonreceipt of chemotherapy matched closely on race, sex, year of diagnosis, and SEER site categories. Younger patients and those with low comorbidity scores were more likely to have started chemotherapy than those who were older and sicker; females were slightly more likely than males to have never received chemotherapy.

Physician characteristics and the numbers of patients attributed to physicians with each characteristic are shown in 

. Oncologists were primarily male (77.4%) and in group practices (61.2%). Physicians in small independent practices were significantly more likely to administer chemotherapy during the last 30 days of life (Table 2). Adjusting for all patient and provider covariates, the predicted probability of receiving chemotherapy in the last 30 days of life was 1.4-fold greater (0.56) for patients receiving care in small independent practices relative to those seeing oncologists in academic centers (0.40; 

). Predicted probabilities for patients receiving care in group practices, government facilities, hospitals, and other types of practice, were 0.55, 0.46, 0.42, and 0.52, respectively. Female oncologists were significantly less likely to administer chemotherapy at end of life (

 <.001). Magnitude and direction of predictors were not meaningfully changed when analyses were repeated for patients of physicians treating at least 10 patients.

Adjusting for other patient and physician characteristics, the predicted probabilities of receiving chemotherapy in the last 30 days of life were lower for blacks, women, and those 75 years or older, diagnosed in 2005 or 2006 (vs 1999), for whom 2 or more months had elapsed since diagnosis, and with comorbidity scores of 3 to 5 or 6 to 8. Even within these subgroups, at least one-third would have been predicted to receive chemotherapy in the last 30 days of life (Table 3). Marital status and zip code—level median income categories were not significant predictors of chemotherapy receipt at end of life.

Overall, the model explained 28.9% of variation in chemotherapy use in the last 30 days of life among patients who were being treated for aNSCLC. Geographic location of care was a significant fixed-effects parameter (

 <.001) and accounted for 16% of variation in chemotherapy use at the end of life. Patient characteristics accounted for 5.8%, while physician characteristics accounted for 7.1% of explained variation in chemotherapy use in the last 30 days of life.

Advanced cancer is emotionally and physically taxing and can cause a great deal of suffering. Although patients are concerned about both quality and quantity of life,

 most prefer not to undergo aggressive treatment at life’s end,

 making it particularly important that they understand expected clinical outcomes from treatment. Yet, Weeks et al found that 69% of patients with advanced lung cancer believed that they had some chance of cure.

 Physician guidance regarding goals of care, prospective benefits of therapy, and helping to determine whether benefits are likely to outweigh expected side effects are critical components of quality care.

Physician prognostication is inexact, however, and variations in practice may reflect this uncertainty to some extent. Healthcare practices are known to vary—sometimes substantially&mdash;by geographic region or hospital

; few studies have examined whether physician characteristics are associated with variation in difficult clinical decisions, such as whether to continue therapy in the face of declining health.

Although the optimal rate of chemotherapy use at the end of life is unknown, our data suggest that nonclinical factors may strongly influence treatment decisions. In our study, 43% of chemotherapy recipients received final doses in the 30 days prior to death. Patients were much more likely to receive late chemotherapy if their physician was in a small independent practice or in a group practice. Of note, physicians in these types of practice were responsible for the care of almost three-fourths of all patients with aNSCLC. Since 2008, the ranks of community oncologists have dwindled,

and there has been a migration toward larger group practices.

Chemotherapy use at the end of life has trended downward during this same period, but it is not known whether these events are correlated or coincidental.

The data do not allow us to conclude why the type of practice is associated with chemotherapy use at the end of life. Many factors are likely to be associated, including differences in practice style—such as attitudes toward aggressive treatment, perceptions of the benefit of treatment, and the desire to provide hope to patients&mdash;and financial incentives for providing more treatment. Physicians in different types of practices may also see patients with diverse expectations and preferences for care, perhaps because patients seeking more aggressive treatment self-select physicians willing to provide that care.

Even for patients desiring aggressive treatments, physicians have a duty to provide treatment only to those who may reasonably be expected to benefit. Increased age and significant comorbid illness decrease the already limited benefits of late chemotherapy; our findings of there being a lower likelihood of these populations initiating chemotherapy suggest that appropriate clinical factors are playing some role in treatment decisions. Aggressive treatment also comes at great financial expense. Treatment of stage 4 aNSCLC is associated with particularly low value: $1.19 million per year of life saved.

 Some commentators envision a “geriatric blast for oncologists” as more Baby Boomers become Medicare-eligible,

 but humanistic and financial imperatives underline a need to reevaluate aggressive care for advanced cancers.

Our study confirms aspects of others’ work—generally conducted in a more heterogeneous group of cancers&mdash;while adding several important dimensions to the literature. Earle found that 15.7% of patients who start chemotherapy received a dose within 14 days of death.

 We confirm even higher rates in the last 30 days of life, specifically among those with aNSCLC. We also reaffirm geographic variation previously noted.

Our findings that physician characteristics predict patterns of care for aNSCLC patients at the end of life contribute a unique dimension to existing literature. Setoguchi, studying quality indicators in end-of-life care for lung, colorectal, breast, and prostate cancers in New Jersey (a SEER site with high treatment rates at the end of life

) and Pennsylvania (not in SEER, average end-of-life rates) noted that oncologists in small group practices were more likely to administer chemotherapy and less likely to initiate hospice than those in large groups.

 Our geographically broader study examined a specific cancer in advanced stage that responds to chemotherapy only to a limited extent. Controlling for geographic variation, we nonetheless found meaningfully higher rates in small independent practices and among male physicians. As others have noted, we found that chemotherapy receipt within 30 days of death decreased in 2005, corresponding with the substantial decline in reimbursement by Medicare for providing outpatient chemotherapy, as well as FDA approval of an oral therapy (erlotinib), not included in our dataset.

This study has several limitations. Retro-spective analysis of administrative data does not allow us to distinguish between aggressive treatment that is patient-driven from that which is physician-driven. It is possible that patients desiring aggressive treatment seek out physicians amenable to their demands, although this would not invalidate the observation that oncologists treating at higher rates gravitate to small independent practices. As our data were limited to Medicare patients living within SEER regions, and analysis was limited to 1999 to 2006 diagnoses, results may not be generalizable to other populations or time periods. Data sources limited the availability of physician demographics, practice characteristics, and the ability to distinguish among practices with more than 2 physicians. That so few measures explained more variation (7.1%) than the robust demographic data available for patients (5.8%) suggests that other physician characteristics may warrant exploration; parameters of interest might include physician wealth, marital status, and race, practice payer mix, and training environment. Assignment of responsibility to the last oncologist seen may incorrectly attribute final doses of chemotherapy, although this is unlikely to introduce bias regarding type of practice. Our study design did not permit inclusion of oral chemotherapeutic agents (eg, erlotinib), possibly used by a subset of patients during later parts of the observation period, but whose inclusion could only have increased treatment rates near death.

The disconnect between how patients report preferring care at the end of life and how they actually die may have any number of causes, including clinical uncertainty, poor prognostication, incomplete sharing of information with patients, misguided optimism, or physicians’ failure to explore patients’ preferences. Poor communication of information is notorious and pervasive. When two-thirds of patients with stage 4 lung cancer are unaware that chemotherapy is unlikely to cure their cancer and, therefore, they do not know they are approaching the end of life,

 surely fewer know that late aggressive therapy may actually foreshorten life.

Improved communication and early incorporation of palliative care can lead to care more consistent with patients’ goals. Early enrollment in palliative care is associated with a significant decrease in receipt of chemotherapy close to death when chemotherapy’s side effects outweigh any potential benefit.

 Better tools are needed to help oncologists determine how to communicate prognoses in ways that patients can understand, as well as how best to partner with patients in shared decision making when a prognosis is poor. Together, these would help determine when to advise a patient to avoid the risks and discomforts of chemotherapy, and instead enjoy what quality can be had during the limited remaining life.

Prognosticative limitations notwithstanding, variable rates of late chemotherapy receipt signify inconsistency in how cancer is treated as death nears. As variations in practice are also a cost driver, evidence of variation between physicians suggests the need to improve physician acceptance of the responsibility to more judiciously steward resources or, failing that, institute policies and practice guidelines to minimize variations in care.

When caring for people with advanced disease, an important aim of medicine includes helping patients experience death on their own terms. The present study provides some support for the common and long-held suspicion that our healthcare system may not always guide patients toward the best choices. Future efforts to improve the experiences of patients with advanced disease may be dampened by the extent to which variation and potential overtreatment are due to the unintended and untoward effects of forces influencing physician decisions. For the time being, it is important for patients to be aware that characteristics of their physician and where they receive care might strongly influence the care they receive.&ensp;

Articles

Patients receiving care for advanced non—small cell lung cancer in small, independent oncology practices are more likely to receive chemotherapy in the last 30 days of life.

Physician Variation in Lung Cancer Treatment at the End of Life

To assess the feasibility and validity of developing electronic clinical quality measures (eCQMs) of cancer care quality from existing metrics, using electronic health records, administrative, and cancer registry data.

Retrospective comparison of quality indicators using chart abstracted versus electronically available data from multiple sources.

We compared the sensitivity and specificity of eCQMs created from structured data from electronic health records (EHRs) linked to administrative and cancer registry data to data abstracted from patients’ electronic health records. Twenty-nine measures of care were assessed in 15,394 patients with either incident lung or prostate cancer from 2007 and 2008, respectively, and who were treated in the Veteran’s Health Administration (VHA).

It was feasible to develop eCQMs for 11 of 18 (61%) lung cancer measures, 4 (22%) of which were considered to be valid measures of the care constructs. Among prostate cancer measures, 6 of 11 (55%) were feasible, and 4 (36%) were both feasible and valid. Of the 29 metrics, data was available to create eCQMs for 17 (59%) cancer care metrics, and 8 (28%) were considered valid.

In a large integrated healthcare system with nationally standardized electronic health records, administrative, and cancer registry data, 28% of cancer quality measures developed for chart abstraction could be translated into valid eCQMs. These results raise much concern about the development of electronic clinical quality measures for cancer care, particularly in healthcare environments where data are disparate in both form and location.

		Only 28% of cancer quality measures could be translated into electronic clinical quality metrics, even in the largest integrated healthcare system in the United States with nationally standardized electronic data repositories.

		These results raise concern about the use of electronic clinical quality measures for cancer care, particularly outside integrated healthcare environments, and suggest that it may be premature to think that meaningful measures of cancer care delivery can be developed for widespread use across electronic health record platforms.

		Current health data systems are inadequate to support meaningful electronic quality measurement in cancer care.

As the healthcare system moves toward a paradigm that rewards “value” in healthcare, or the maximizing of quality while minimizing costs, accurate measurement of the quality of healthcare delivery has taken center stage in reform efforts.

 Currently, most quality measures rely on for data acquisition chart abstraction, which is considered the gold standard. While chart abstraction allows access to all types of medical data, clinical notes in particular, it is time-consuming and prohibitively expensive to permit broad and timely measurement of care quality.

 It has long been hoped that electronic measurement of care delivery would open the door to timely, meaningful, and actionable information for many stakeholders.

 Electronic health records (EHRs), now adopted by more than 70% of physicians, and health information exchanges are 2 key components of the visions of data interoperability.

 This is coupled with technical developments, like natural language processing (NLP) and structured clinical templates, which are expected to facilitate computerized access to the content of free text that underlies much of primary clinical documentation, and

 payment reforms, like accountable care organizations, which encourage linkage of healthcare data silos.

7 While human chart abstractors are able to bridge data silos and interpret the full range of available data sources, it is still unclear if the above reforms will make electronic clinical quality measures (eCQMs) and measurement feasible. Many studies have illustrated wide gaps in agreement among electronically available data sources and between these sources and chart abstraction.

 Furthermore, eCQMs that are solely based on administrative data have been criticized for failing to measure what matters.

 Cancer care, additionally, is especially challenging to measure, in part because it relies on the multifactorial process of staging and because care is typically delivered across multiple specialties and care settings.

Large integrated healthcare systems, like the Veterans Health Administration (VHA), having already achieved national data integration across all aspects of care delivery, with a common EHR and national, standardized, electronically available data warehouses, offer an opportunity to examine the feasibility and validity of using available electronic data to measure cancer care quality.

 Recently, the VHA performed quality assessments of care for lung and prostate cancer patients using manual chart abstraction.

We hypothesized that the data needed to measure quality performance was available in the VHA’s electronic clinical, administrative, and cancer registry repositories and could be used to create eCQMs from existing quality measures.

This was a retrospective, national VHA cohort study sponsored by the VHA Office of Analytics and Business Intelligence. The VHA Greater Los Angeles Institutional Review Board approved this study.

VHA expert panels reviewed the relevant literature and proposed quality measures—using the RAND/ UCLA modified Delphi technique, for national assessments of the quality of care for lung cancer and prostate cancer diagnosed in 2007 and 2008, respectively&mdash;to be performed using data abstracted from patients’ EHRs.

 There were 29 quality metrics (18 for lung cancer and 11 for prostate cancer) spanning the continuum of care (see Appendix for more detailed descriptions). Many measures replicated, or were similar to, metrics endorsed by non-VHA organizations (eAppendix). Twenty measures addressed diagnosis, treatment, and management; 5 addressed supportive care; and 4 addressed end-of-life care. Twenty-six measures were patient level, 2 were visit level, and 1 was medical center level.

Data necessary for 29 measures of the quality of lung and prostate cancer care were abstracted from patients’ VHA Computerized Patient Record System (CPRS) medical records by External Peer Review Program (EPRP) nurse abstractors conducting the national VHA assessments of the quality of cancer care (chart abstraction cohort). Using an abstraction tool specifically developed for the quality assessments, EPRP abstractors systematically abstracted the data necessary to score the quality indicators. Case-level quality indicator and timeliness results were provided to VAMCs for review and correction, and the final results were calculated based on these field-corrected data.

Data for 29 eCQMs were obtained from extracts from the VHA’s EHR through the Corporate Data Warehouse (CDW) and Decision Support System (DSS), both of which are national, standardized, and near real-time clinical and administrative data repositories; and from the VHA Central Cancer Registry (VACCR). The eCQM data set was created by linking data from the VACCR, CDW and DSS. This included inpatient and outpatient encounter and procedure coding, as well as pharmacy, lab, radiology, allergy, problem list, and vital sign data.

In both the chart abstraction and eCQM cohorts, case identification criteria were identical: all newly diagnosed, pathologically confirmed cases of lung cancer during 2007 (n = 8125) and prostate cancer during 2008 (n = 12,572) that were reported to the VACCR. Patients were excluded if the pathologic diagnosis was not identified in the EHR at the index facility that reported the case to the registry (lung, 1297; prostate, 489), a situation that could occur if patients were treated at more than 1 facility.

Patients were excluded from the chart abstraction cohorts if they were diagnosed at autopsy; death occurred in 30 days or less after cancer diagnosis; hospice enrollment occurred 30 days or less after cancer diagnosis (lung, 947; prostate, 16); they were enrolled if they had a pre-existing or concurrent diagnosis of metastatic cancer other than lung or prostate cancer (lung, 540; prostate, 627); they were enrolled in a cancer clinical trial (lung, 57; prostate, 147); there was documentation of comfort measures only in a hospital discharge summary or nursing home note 30 days or less after cancer diagnosis (lung, 91; prostate, 14); or documentation of life expectancy of 6 months or less in their Problem List at the time of diagnosis (lung, 39; prostate, 4). The resulting lung cancer and prostate cancer abstraction cohorts included 4865 and 11,211 patients, respectively.

Patients were excluded from the lung and prostate cancer eCQM cohorts if they died within 60 days of diagnosis (lung, 1482; prostate, 674); if there was no pathologic confirmation of disease according to the VACCR (lung, 419; prostate, 49); or if there were less than 2 encounters with an 

International Classificiation of Diseases, Ninth Revision

 (ICD-9) diagnosis code for prostate or lung cancer (lung, 229; prostate, 8). The eCQM cohort included 5995 patients with lung cancer and 11,700 patients with prostate cancer.

The final validation cohort included patients common to both the lung and prostate abstraction and the eCQM cohorts, n = 4865 and n = 10,529, respectively.

Adaptation of quality measures for eCQMs.

 Using the linked eCQM data set, we identified the elements specified by each quality indicator and, where more than 1 option existed, determined the optimal specification iteratively. Two measures, “No adjuvant chemotherapy for stage IA NSCLC” and “No radiation therapy for resected stage I, II NSCLC,” were excluded from analysis prior to eCQM creation because adherence, by chart abstraction, was greater than 99% with no variation seen across facilities.

eCQMs were considered feasible if electronically available data could specify each aspect of the measure. Validity was evaluated by comparing the sensitivity and specificity of the denominator and numerator for each measure, using chart abstraction as the gold standard, and by comparing overall pass rates between the electronic and the chart-abstracted versions of the measures. Although there is no accepted cut-point for numerator and denominator sensitivity and specificity in determining measure validity, nor an accepted level of agreement between pass rates that marks validity, we considered at least 80% specificity for both the denominator and the numerator (compared with chart abstraction data) as valid for an eCQM. However, quality measures with less than 80% specificity or sensitivity were assessed individually and considered likely valid as eCQMs if internal validity could be supported—even if results from measure use might not be comparable across data collection methodologies. Analyses were performed using SAS statistical software, version 9.1 (Cary, North Carolina).

Patient demographic characteristics were similar to national trends in lung and prostate cancer (

). The distribution of patients by AJCC stage (lung) or D’Amico risk group classification (prostate) were similar in both the chart abstraction cohort and the VACCR, with 25.5%, 7.8%, 28%, and 34.2% of lung cancer patients with Stages I, II, III, and IV, respectively.

 Among men with prostate cancer in the chart abstraction cohort, 24.5%, 34.8%, and 26.3% had low-, intermediate-, and high-risk disease, respectively. The overall percent agreement in staging classification between the VACCR and chart abstraction was 65% (95% CI, 63-67) among lung cancer cases and 75% (95% CI, 74-77) for D’Amico risk classification in prostate cancer cases.

 Using the VHA EHR, administrative, and VACCR data, it was possible to specify the necessary data elements for 17 of 29 (59%) of the eCQMs (

). This included 6 of the 11 prostate cancer quality measures (55%) and 11 of 18 lung cancer quality measures (61%). In some cases, to accommodate available electronic data while attempting to preserve the intent and face validity of the measure, changes were required of the quality measure definitions. For example, the numerator of “surgical node sampling” in lung cancer treatment was modified to include cases based on the number of lymph nodes sampled instead of the number of lymph node stations that were sampled. While the number of nodes has value in determining staging, and may correlate with the number of lymph node stations, it is not an exact match and limits the ability to compare quality scores using the 2 different approaches.

Measures for which it was not feasible to create eCQMs are listed in 

 Eight of the 17 measures (47%) that were feasible as eCQMs, which was 28% (8/29) of the initial set of quality measures, were found to be valid, (Table 2, eAppendix A). All of the measures examined processes of care at the patient level. The specificity of denominators for the valid eCQMs ranged from 84% to 98% (Table 2). In 7 of the valid eCQMs, the specificity of the denominator was equal to or greater than 90%; in 1 case it was equal to or greater than 80%. The sensitivity of the numerators for the valid eCQMs ranged from 84% to 100%. In 4 of the valid eCQMs, the sensitivity was equal to or greather than 90% and in 4 it was equal to or greather than 80%. Pass rates among valid eCQMs ranged from 61% to 99% for quality measures, based on chart abstraction, and from 65% to 100% for the same eCQMs. The difference in the pass rate between chart abstraction and eCQMs ranged from 1% to 20% for valid measures.

Four measures (surgical node sampling and outpatient screening for pain in both advanced disease and prior to death, with PSA monitoring after treatment) that failed our minimum sensitivity and specificity criteria were nevertheless considered “likely valid” based on individual consideration. Data definitions varied slightly between chart abstraction and eCQM cohorts; this, in conjunction with the expected use of these measures in a context where local data validation would take place, led us to believe that they would likely be valid and to recommend their further use. Had we included these 4 measures as valid in this analysis, the total number of valid measures would have been 12 of the 29 measures considered (41%).

For example, 2 measures of outpatient screening for pain were visit-level measures where the results assessed using the 2 different methods of data capture were not necessarily comparable. However, since all patients with visits were eligible for these quality measures, the measures were thought to be valid with either approach.Another example: the measure of PSA monitoring after treatment, which was likely performed less well as an eCQM, resulting in the inclusion of ineligible patients. This was due, in part, to the definition of active surveillance which, in chart abstraction, required affirmative documentation of this, but was defined as a lack of other treatments in the electronically available data. Five feasible measures were not valid (Table 2). For 3 of these, this was due to low sensitivity and specific, and poor pass rate agreement (adjuvant chemotherapy for resected stage II or III NSCLC, platinum-based doublet chemotherapy for stage IV NSCLC, and repeat biopsy for men on active surveillance). An additional measure of chemotherapy administration (platinum-based doublet chemotherapy for SCLC) was not considered valid, despite favorable sensitivity and specificity, because of a relatively large difference in pass rate, generalized concern about the validity of determining receipt of chemotherapy-raised by the other 3 chemotherapy-related measures, and limited utility, demonstrated by a nearly perfect pass rate by chart abstraction data.

Of the 12 measures that were feasible and valid or likely valid as eCQMs, none would have been feasible without data from the VACCR. The reverse was also true: none of the measures could have been specified with VACCR data alone.

Critics have pointed to a lack of value in eCQMs that were designed around available data, as opposed to clinically important data, or as relying too much on burdensome structured data entry by providers.

 In this study, we validated eCQMs for lung and prostate cancer that were adapted from existing quality measures. Our finding that 28% of lung and prostate cancer quality measures, which were designed without regard to electronic abstraction, could be translated into eCQMs using only available data, underscores the opportunity in electronic quality measurement and the effectiveness of VHA’s data management efforts. However, the fact that 72% of the measures could not be validated as eCQMs (in one of the most integrated and data-rich environments in healthcare) is a reminder of the limits of current healthcare data systems to facilitate the measurement of quality of care. As multiple stakeholders look toward EHR-derived data for the next generation of quality measurement, our results highlight the difficulty of using newly available electronic data and existing metrics to assess quality, particularly in nonintegrated healthcare systems.

At present, there is little consensus or guidance on appropriate validation criteria for eCQMs, although some effort is being made in this area.

 In this study, we used 4 criteria when considering the likely validity of eCQMs: (1) the sensitivity and specificity of the denominator, placing particular emphasis on a high specificity to favor inclusion of only true eligible cases in the denominator; (2) the sensitivity and specificity of the numerator, placing particular emphasis on a high sensitivity to reduce the risk of excluding appropriately delivered care; (3) the agreement between chart abstracted and eCQM pass rates as another global comparator of measure performance; and (4) the clinical context of measure use as determined by the type of use (eg, quality improvement vs accountability), the utility of the measure (as determined, in part, by the overall pass rate and the resultant opportunity for improvement), and other clinical estimations of the measure’s usefulness.

All of the eCQMs we evaluated relied on data from the VACCR and, therefore, on trained cancer registry chart abstractors, underscoring the importance of disease staging for quality measurement in oncology. Natural language processing and synoptic reporting templates, like that currently offered by the College of American Pathologists, may reduce or eliminate the need to chart abstract information contained in free text form, like disease stage and operative reports, however, both still rely on accurate primary clinical documentation.

 The percent agreement between trained quality abstractors and trained cancer registrars in this study reinforces the need to validate data from any source. An alternate way forward is wider use of electronically available and exchangeable cancer registry data. Registry-EHR interfaces and incorporating cancer registries into quality measurement programs is a goal of the American College of Surgeons’ Rapid Quality Reporting System.

 Data aggregation programs, like the American Society of Clinical Oncology’s CancerLinQ™, may provide another avenue for access to the data in registries and in other disparate locations and forms.

Measurement can drive rapid changes in healthcare, but there is no guarantee of improvement, particularly if, in addition to validation, concrete plans for re-evaluation of measures are not built into their use.

 In this study, it took 2 years to develop and validate the eCQMs, and they have yet to be implemented in an ongoing quality improvement program. Furthermore, nearly 10% of the measures evaluated in this study (“3D-CRT,” “No adjuvant chemotherapy for stage IA NSCLC,” and “No radiation therapy for resected stage I, II NSCLC”) were “topped-out” with no opportunity for improvement. Given the time and other resources involved in developing measures, there is a real risk that measures will fall behind the corpus of medical knowledge and have the unintended consequence of serving not as stimuli for quality, but as barriers that physicians must overcome (perhaps at risk to one’s finances and reputation) in order to deliver the best care possible.

There are several limitations to this study. Our study may not be generalizable to all cancer types or healthcare settings. As described above, some quality measures were altered so that they could be specified as eCQMs, and this limited our ability to validate them. VHA EHR data on clinical encounters is largely based on ICD-9 and Clinical Procedure Terminology coding, and coding accuracy at the VHA is likely lower than in the fee-for-service setting. Also, some of the patients in this study received parts of their cancer care outside of the VHA. In these cases, documentation is sometimes not visible to electronic abstraction, likely disproportionately decreasing the pass rates for eCQMs versus chart abstraction.

Using quality measures that were designed without consideration of electronic data availability, electronic data in the VHA is comprehensive and accurate enough for use in electronic quality measurement of care in lung and prostate cancer. However, more than half of the metrics could not be translated into electronic form, and abstraction relied in part on VACCR data. Automated data extraction and quality measurement are likely to increase in the coming years, although this report suggests significant barriers to using existing clinical data, even in a large integrated healthcare system. Furthermore, without validation studies, quality metrics based on EHR-derived clinical and administrative data run the risk of distorting, rather than enhancing, the assessment of quality in oncologic care.

 From the VA Greater Los Angeles Healthcare System (AA) Urology, Informatics, HSR&D (JBS) and Oncology (JM), Los Angeles, CA; University of Michigan, VA Ann Arbor Healthcare System, Urology, HSR&D, Ann Arbor, MI (TAS); VA (former), Washington, DC (DO); Kaiser Permanente, Los Angeles, CA (JR); and University of California, Los Angeles, CA (CSS).

 This study was funded by the Veterans Affairs Office of Analytics and Business Intelligence. This study was supported in part by the UCLA Career Development Program in Cancer Prevention and Control Research R25 Grant CA 087949-13 (JBS).

The authors report no relationship or financial interest with any entity that would pose a conflict of interest to the topic of this paper.

Concept and design (JBS, DO, JM, AA, JR, CSS); acquisition of data (JBS, JM, JR, CSS); analysis and interpretation of data (JBS, TAS, JM, AA, JR, CSS); drafting of the manuscript (JBS, TAS, JM); critical revision of the manuscript for important intellectual content (JBS, TAS, JM); statistical analysis (JBS, JM, AA, JR); provision of study materials or patients (); obtaining funding (JBS, DO, JM); administrative, technical, or logistic support (JBS, DO); and supervision (JBS, JM).

 Jeremy Shelton, Veterans Affairs - Surgery, 13301 Wilshire Blvd, Bldg 500, Los Angeles, CA 90073. E-mail: 

1. Porter ME. What is value in health care? 

2. Gabow P, HGKG. Marshaling leadership for high-value health care: an institute of medicine discussion paper. 

Even in a fully integrated healthcare system, only 28% of cancer quality measures could be validated by using electronically available data.

Jennifer Malin, MD, PhD

Articles

Physician Variation in Lung Cancer Treatment at the End of Life

Validating Electronic Cancer Quality Measures at Veterans Health Administration