A health insurance claims-based risk assessment tool to predict patients’ first severe chronic obstructive pulmonary disease exacerbation has been developed and validated.
Objectives: To develop and validate a predictive model for first severe chronic obstructive pulmonary disease (COPD) exacerbation using health insurance claims data and to validate the risk measure of controller medication to total COPD treatment (controller and rescue) ratio (CTR).
Study Design: A predictive model was developed and validated in 2 managed care databases: Truven Health MarketScan database and Reliant Medical Group database. This secondary analysis assessed risk factors, including CTR, during the baseline period (Year 1) to predict risk of severe exacerbation in the at-risk period (Year 2).
Methods: Patients with COPD who were 40 years or older and who had at least 1 COPD medication dispensed during the year following COPD diagnosis were included. Subjects with severe exacerbations in the baseline year were excluded. Risk factors in the baseline period were included as potential predictors in multivariate analysis. Performance was evaluated using C-statistics.
Results: The analysis included 223,824 patients. The greatest risk factors for first severe exacerbation were advanced age, chronic oxygen therapy usage, COPD diagnosis type, dispensing of 4 or more canisters of rescue medication, and having 2 or more moderate exacerbations. A CTR of 0.3 or greater was associated with a 14% lower risk of severe exacerbation. The model performed well with C-statistics, ranging from 0.711 to 0.714.
Conclusions: This claims-based risk model can predict the likelihood of first severe COPD exacerbation. The CTR could also potentially be used to target populations at greatest risk for severe exacerbations. This could be relevant for providers and payers in approaches to prevent severe exacerbations and reduce costs.
Am J Manag Care. 2018;24(2):e45-e53Takeaway Points
Chronic obstructive pulmonary disease (COPD) is the third leading cause of death in the United States, with more than 147,000 patients dying because of it in 2014.1 COPD poses a substantial economic burden on patients, healthcare systems, and society. In the United States, $32.1 billion in medical costs were associated with COPD in 2010, with costs projected to rise to $49.0 billion by 2020.2
A major portion of COPD-related healthcare costs are attributable to exacerbations (sustained worsening of COPD from stable state, beyond normal day-to-day variations).3,4 Exacerbations are acute in onset and warrant additional treatment.3,4 Increased frequency of exacerbations leads to more hospital and emergency department (ED) visits.5 In 2009, approximately 740,000 hospitalizations were related to COPD.6 Hospitalization costs represent 85% of direct medical costs associated with COPD and are largely associated with exacerbations.7,8 Based on a Healthcare Cost and Utilization Project analysis, Perera et al showed that severe exacerbations, defined as hospitalizations for acute COPD exacerbations, were associated with increased aggregate costs, rising from $2.96 billion in 2006 to $3.47 billion in 2010.9 Furthermore, in a retrospective analysis of the Thompson Reuters MarketScan administrative database, Yu et al reported that COPD-related costs for patients with severe exacerbations were $7014 per 90 days compared with $658 per 90 days for patients with no exacerbations, with 83% of the costs associated with inpatient expenses.10 In another retrospective study, Abudagga et al reported that the costs associated with severe exacerbations were $18,120 and increased with each additional prior exacerbation.11
Given the high costs associated with COPD exacerbations, it is important to identify patients at risk of exacerbation to target them for early treatment and improved prevention. Prior history of exacerbations is among the most important risk factors for the development of subsequent exacerbations.12 Previous clinical studies have identified additional risk factors for COPD exacerbations, with several groups developing exacerbation risk models.13-22 However, risk assessment models for COPD exacerbations utilizing readily accessible claims data that are available to health plans and healthcare groups, such as national quality-of-care organizations, are not available.
Identifying patients at risk for their first severe exacerbation is critical to target early and effective treatment, potentially slowing progression and associated cost increases. The present study was a secondary analysis from a wider study in which a medication ratio using pharmacy claims and a risk model were developed and validated. The primary results of this study have been published.23 In this secondary analysis, we aimed to construct and validate a predictive model of first severe COPD exacerbation event using health insurance claims, a source of data readily available to health plans and most quality-of-care organizations. This secondary analysis aimed to further validate the ability of the controller medication to total COPD treatment (controller and rescue) ratio (CTR) to predict first severe COPD exacerbation. The CTR is similar to the current Healthcare Effectiveness Data and Information Set (HEDIS) measure in asthma, the asthma medication ratio, and could have the potential to be used in a similar capacity.24
The primary objectives of this study have been reported elsewhere.23 There were 2 objectives of the present analysis: 1) construction and validation of a risk model for first severe COPD exacerbation using patient demographics, comorbid conditions, and COPD treatment claims data as covariates; and 2) validation of the CTR as a measure of severe COPD exacerbation risk. The CTR is the ratio of total controller medications (inhaled corticosteroids [ICS], long‑acting β2-adrenergic agonists [LABA], ICS + LABA, long‑acting muscarinic antagonists, phosphodiesterase-4 inhibitors, and methylxanthines) to total controller plus rescue medications (short-acting β-agonists or short-acting muscarinic antagonists [SAMA]).
This retrospective study (GlaxoSmithKline study #HO-11-732) used the Truven MarketScan Commercial Claims and Encounter Database and Medicare Supplemental and Coordination of Benefits database (MarketScan database, Truven Health Analytics; Ann Arbor, Michigan) from 2006 to 2011 to construct and validate a risk model of prediction of first severe COPD exacerbation and to test the CTR measure. The Reliant Medical Group (Reliant database, Reliant Medical Group; Worcester, Massachusetts) was used for cross-validation.
The study start date (index date) was the second encounter with a COPD diagnosis (defined in study population section) within a year. Risk factors were assessed during the baseline period, which spanned 1 year, starting at the index date. This was followed by a 1‑year at-risk period during which the occurrence of severe COPD exacerbations, the primary endpoint, was assessed (Figure).
Patients with COPD who were 40 years and older were included in the study. The index date of COPD status was defined as the second outpatient and/or ED encounter with a COPD diagnosis (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] codes 491.x [chronic bronchitis], 492.x [emphysema], or 496 [chronic airway obstruction]) within 1 year. Patients were also required to have at least 1 dispensing for an inhaled COPD medication or oral theophylline during the year following the index date and to have had continuous insurance eligibility (no more than 45 days without coverage) for at least 2 years following the index date.
Patients were excluded if they had a severe COPD exacerbation, defined as a COPD-related hospitalization during the baseline period, or at least 1 claim with an ICD-9-CM diagnosis code for cancer (except cancer types that are typically considered to have little effect on lung function [eg, breast, prostate, or skin cancer except melanoma]) or other non-COPD lung disease during the baseline period.
The MarketScan database was used for model development and validation. This database includes health insurance claims data from 2006 to 2011, representing approximately 18 million covered individuals, and contains information on enrollment history and claims for medical (provider and institutional) and pharmacy services. The Reliant database was used for external validation. It includes longitudinal member-linked medical claims (physician and facility), pharmacy claims, enrollment records, laboratory results, and electronic health record information.
The primary endpoint of this study was the occurrence of a severe COPD exacerbation during the at-risk period (Year 2). A severe COPD exacerbation was defined as a hospitalization with either a primary diagnosis of COPD (excluding obstructive chronic bronchitis without exacerbation; ICD-9-CM code 491.20) or a secondary diagnosis for COPD with a primary diagnosis of respiratory failure (ICD-9-CM codes 518.81, 518.82, or 518.84). The choice of a 1-year window for the baseline and at-risk periods for the base case scenario made it possible to account for the seasonality of COPD exacerbations. Yet, given that risk factors were evaluated in the year prior to the at-risk period, they may not accurately reflect risk factors during the totality of the at-risk period. For example, the CTR may reflect more reliably the COPD medications patients received at the beginning of the at-risk period than those received toward the end of the at-risk period. As such, sensitivity analyses were conducted that limited the at-risk period to 90 days or 180 days following the baseline year (Year 1).
Candidate risk factors were assessed during the baseline period and used as predictors for severe exacerbation risk in the following year. Candidate risk factors were gender, age, region, insurance plan, H1N1 flu season (October 2009 to May 2010), pulmonologist visit, county characteristics (altitude; number of pulmonologists per 100,000 inhabitants; number of hospitals per 100,000 inhabitants; proportion of households below the low-income margin; median household income; proportion of patients without health insurance; proportions of high school dropouts and college graduates; urban/suburban/rural localization; and proportion of white, black, Hispanic, and Asian patients, derived from the Area Resource File25), type of COPD diagnosis, exacerbation history (moderate only [ie, defined as outpatient treated or ED visit for COPD with a dispensing for an oral corticosteroid within 7 days]), COPD medications (based on at least 1 dispensing), concomitant medications, procedures (flu and pneumococcal vaccines, use of chronic oxygen therapy, nebulizer, and spirometry; all based on Current Procedural Terminology and Healthcare Common Procedure Coding System codes), and comorbidities based on ICD-9-CM codes.26 Based on univariate associations among risk factors and the probability of an exacerbation in the at-risk period, risk factors were excluded if P >.1. P values were calculated using χ2 tests for discrete variables and Wilcoxon tests for continuous variables. Risk factors present in less than 0.5% of the sample were excluded.
The choice of factors for the risk model was based on bootstrap resampling in combination with stepwise variable selection with logistic regression modeling, using a random sample of the MarketScan sample (development sample).27 A total of 1000 bootstrap samples were drawn; each was a random sampling with replacement from the original development sample, and each bootstrap replication was done using a different bootstrap sample. Within each bootstrap replication, all candidate risk factors were included, then stepwise elimination of risk factors was performed using the Akaike information criterion to identify predictors of exacerbations in the final model for the replication. Risk factors were chosen if they were selected in: 1) 100% of the 1000 bootstrap replications for the modeling of severe exacerbation or 2) at least 90% of the bootstrap replications for severe exacerbation and in 100% of the bootstrap replications for moderate or any type of exacerbations. CTR, which was defined as the number of canisters dispensed (converted to 30-day equivalent) for controller medication divided by the number of canisters dispensed for controller plus number of canisters of rescue medication (ie, SAMA, short‑acting β2-adrenergic agonists [SABA], SAMA + SABA), was included in the final risk model. In order to assess the CTR as a predictive measure of first severe exacerbation, various ratio thresholds from at least 0.1 to at least 0.9 were tested. A cutoff of 0.3 was found to be the cutoff associated with the largest statistically significant risk reduction for severe exacerbations and has been shown to be the only cutoff associated with a statistically significantly reduced risk of any, moderate, and severe exacerbations in the previously published primary objectives of this study.23 Therefore, this ratio was used in the risk model of first severe COPD exacerbations.
The risk model was developed using the development sample, validated using the other random sample of the MarketScan data (validation sample), and cross-validated using the Reliant Medical Group data (cross-validation sample). Performance of the risk model was evaluated using C-statistics.28,29 Additional model analyses were performed on selected subpopulations, including those without a history of asthma and differing controller medication usage.
All statistical analyses were performed using SAS version 9.2 or newer (SAS Institute, Inc; Cary, North Carolina).
A total of 223,824 patients without history of severe exacerbations were included in the study, with 111,904 used in the development sample. Of the entire MarketScan sample of patients without history of severe exacerbations, approximately 80% experienced no exacerbations in the at-risk period (Table 1 [part A and part B]). Patients who experienced exacerbations during the at-risk period tended to use more COPD medications and were more likely to use chronic oxygen therapy during the baseline period than patients who did not experience an exacerbation during the at-risk period (P <.01).
Risk Factor Modeling for Severe Exacerbation
A total of 30 risk factors were included in the model; all were statistically significantly associated with severe exacerbations. The 5 risk factors with the largest impact on the probability of developing a severe exacerbation in the at-risk year were advanced age, use of chronic oxygen therapy, use of 4 or more canisters of rescue medication, having at least 2 prior moderate exacerbations, and type of COPD diagnoses received during the baseline period (Table 2 [part A and part B]).
Patients with a CTR of 0.3 or greater had a lower risk of developing severe exacerbations (odds ratio [OR], 0.86; 95% CI, 0.77-0.96) (Table 2). In a subpopulation analysis, among patients with 4 or more fills for controller medications, a CTR of 0.3 or greater was associated with an even lower risk of severe exacerbation (OR, 0.76; 95% CI, 0.64-0.91) (Table 3).
The model performed relatively well in predicting severe exacerbation risk, with C-statistics in the MarketScan development and validation and Reliant cross-validation samples of 0.713, 0.714, and 0.711, respectively (Table 2).
Most risk factors for severe exacerbation identified as most relevant using the 1-year at-risk period (eg, advanced age, use of chronic oxygen therapy, and having at least 2 prior moderate exacerbations in the baseline year) were also identified in sensitivity analyses using either a 90- or 180-day at-risk period (eAppendix Table [eAppendices available at ajmc.com]). Based on C-statistics, the performance of these models was similar to that of the model based on a 1‑year at-risk period.
This report presents the development and validation of a risk model and the testing of the CTR for prediction of first severe COPD exacerbation using health insurance claims information. The main risk factors of first severe COPD exacerbation identified included advanced age, prior moderate exacerbation history, use of rescue medication, use of chronic oxygen therapy, and type of COPD diagnosis. A CTR of 0.3 or greater also was determined to predict first severe COPD exacerbation. The main advantage of the risk model and the CTR measure is that they do not rely on medical records, which are often unavailable to quality-of-care organizations and payers. Unlike previously reported measures, the CTR can be used to specifically assess the risk of first severe exacerbation and may help to identify which patients should be targeted for more intense management so that hospitalizations and costs could be reduced.
This model performed well in predicting the risk of first severe exacerbation at a population level using the MarketScan development and validation samples and the Reliant validation sample. Given the differences between the national MarketScan population and the regional Reliant database, the relatively consistent performance of the risk model in the Reliant population, with a C-statistic value of 0.711, indicates that the model and the CTR could be generalizable to heterogeneous populations.
Prior history of exacerbations is known to be among the most important risk factors for developing subsequent exacerbations.12,18 In an analysis of patients enrolled in the Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints (ECLIPSE) study, history of exacerbation was shown to be the best predictor of exacerbation, regardless of Global Initiative for Chronic Obstructive Lung Disease (GOLD) stage.12 Previous incidence of exacerbations was also found to be an independent predictor of an exacerbation within 24 months (OR, 5.07; 95% CI, 2.55-10.07) in a study of 243 patients in the Netherlands.18 Our study confirmed that prior moderate exacerbations were a risk factor for a subsequent first severe exacerbation; however, a higher CTR value was associated with a reduced risk of first severe exacerbation regardless of exacerbation history. These results suggest that identifying patients at risk for their first severe exacerbation would be critical for targeting early and effective treatment, preventing worsening of the disease, and consequently reducing costs associated with COPD exacerbations.7,8
There is a lack of published literature on risk factors for first severe COPD exacerbations. Müllerova et al evaluated the risk factors for a subcohort of patients with COPD from the ECLIPSE study who had experienced exacerbations requiring hospitalization for the first time during the 3-year study follow-up period.20 Significant risk factors identified were health status, COPD severity (based on forced expiratory volume in 1 second [FEV1]), increasing age, existence of emphysema or asthma, use of home oxygen, and elevated fibrinogen levels.20 Because data on COPD severity or fibrinogen levels were not available in the MarketScan data set, their relevance to exacerbation risk was not considered in this study.
Other studies have focused on the risk of severe exacerbation in general and did not model the risk of first severe COPD exacerbation.13,15,16 Garcia-Aymerich et al considered various clinical, medication compliance, lifestyle, and sociodemographic risk factors for COPD-associated hospitalizations.16 They found that previous COPD-associated hospitalizations, COPD severity (FEV1), under-prescription of long-term chronic oxygen therapy, and smoking were the most relevant risk factors for subsequent COPD-associated hospitalizations.16 In a follow-up study of patients from a clinical trial of tiotropium, Niewoehner et al identified increasing age, COPD severity, prior unscheduled ED visits or hospitalizations for COPD, cardiovascular comorbidities, and use of oral corticosteroids as significant risk factors for COPD-associated hospitalization.13
Although previously published risk models of COPD exacerbations have focused on exacerbation frequency and COPD-associated hospitalizations, rather than the risk of first severe exacerbation, their findings were consistent with those of the current study in that factors such as increasing age and exacerbation history were highly relevant.12,13,15,17-19 Moreover, the performance of the claims-based model compares favorably with these models. In an exacerbation risk model developed by Bertens et al, in which exacerbation history, COPD severity, smoking, and vascular disease history were identified as risk factors, a C-statistics value of 0.66 was reported.18 In a model for frequent exacerbations published by Miravitiles et al, in which chronic mucus hypersecretion, COPD severity, and increasing age were associated risk factors, C-statistics values of 0.601 to 0.655 were obtained.15
Certain methodologic limitations in this study should be noted. First, the identification of exacerbation events was based on definitional algorithms using health insurance claims data that have not been validated. These may have less than 100% sensitivity and specificity and may have misclassification errors for the outcome variables. Second, health insurance claims data are subject to inaccuracies and mistakes. In particular, an ICD-9-CM diagnosis code on a medical claim is not an attestation that the patient has the diagnosis, because the code may represent a rule-out diagnosis or may be recorded incorrectly. This limitation applies to both the exacerbation events and the comorbidities used as risk factors in the models. Additionally, these records do not confirm that patients took the dispensed medication. A third study limitation was that included patients might have experienced severe exacerbations prior to the index date of this study. Finally, the covariates for the risk equations were limited to health insurance claims data only, which lack disease severity information (eg, GOLD stage). Nevertheless, using a subset of the Reliant sample, the performance of the risk model, including GOLD stage, was assessed and showed only a marginal improvement in incremental predictive accuracy.
A claims-based risk model has been developed that can accurately predict the likelihood of first severe COPD exacerbation. Previous outpatient treated exacerbations, advanced age, use of rescue medication and chronic oxygen therapy, and type of COPD were identified as the greatest risk factors. In addition, the CTR, which can be calculated from pharmacy claims data, can predict the likelihood of a first severe COPD exacerbation. The use of this risk model and the CTR may allow policy makers, providers, and payers to identify those patients at highest risk for their first COPD-related hospitalization. Ultimately, the CTR coupled with other HEDIS measures has the potential to improve the management of patients at risk of developing severe COPD exacerbations, thus avoiding worsening of the disease and significant downstream costs.Author Affiliations: GlaxoSmithKline (RHS, AN), Research Triangle Park, NC; Lovelace Clinic Foundation (DWM), Albuquerque, NM; University of Illinois at Chicago (TAL), Chicago, IL; Reliant Medical Group (RR), Worcester, MA; Kaiser Permanente Southern California (MS), San Diego, CA; Groupe d’analyse Ltée (FV, MG-L), Montréal, Québec, Canada; Université du Québec à Montréal (JFPM), Montréal, Québec, Canada; Analysis Group, Inc (MSD), Boston, MA.
Source of Funding: Support for this study was provided by GlaxoSmithKline (GSK) (HO-11-732).
Author Disclosures: Dr Stanford is employed by GSK, the study sponsor, and owns GSK stock. Dr Nag is employed by Shire Pharmaceuticals, was employed by Humana, and owns GSK stock. Dr Lee received consulting money from GSK for his participation in this project. Dr Schatz is a member of a GSK severe asthma advisory board, has received a grant from GSK for a severe asthma database study, and received payment for his involvement in the development of the study concept and methods. Mr Vekeman was and Dr Duh is employed by Analysis Group, which has received research funds from GSK to conduct this and other research. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.
Authorship Information: Concept and design (RHS, AN, DWM, TAL, RR, MS, FV, MG-L, JFPM, MSD); acquisition of data (AN, RR, FV, MSD); analysis and interpretation of data (RHS, AN, DWM, TAL, MS, FV, MG-L, JFPM, MSD); drafting of the manuscript (RHS, DWM); critical revision of the manuscript for important intellectual content (RHS, AN, DWM, TAL, RR, MS, FV, MG-L, MSD); statistical analysis (DWM, FV, MG-L, JFPM); obtaining funding (RHS, AN, MSD); administrative, technical, or logistic support (RHS, FV); and supervision (RHS, AN, FV, MSD).
Address Correspondence to: Richard H. Stanford, PharmD, MS, GlaxoSmithKline, 5 Moore Dr, Research Triangle Park, NC 27709. Email: email@example.com.REFERENCES
1. Kochanek KD, Murphy SL, Xu J, Tejada-Vera B. Deaths: final data for 2014. Natl Vital Stat Rep. 2016;65(4):1-122.
2. Ford ES, Murphy LB, Khavjou O, Giles WH, Holt JB, Croft JB. Total and state-specific medical and absenteeism costs of COPD among adults aged ≥18 years in the United States for 2010 and projections through 2020. Chest. 2015;147(1):31-45. doi: 10.1378/chest.14-0972.
3. Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease. Global Initiative for Chronic Obstructive Lung Disease website. goldcopd.org/global-strategy-diagnosis-management-prevention-copd-2016. Updated December 2015. Accessed January 15, 2017.
4. Burge S, Wedzicha JA. COPD exacerbations: definitions and classifications. Eur Respir J Suppl. 2003;41(suppl):46s-53s.
5. Pasquale MK, Sun SX, Song F, Hartnett HJ, Stemkowski SA. Impact of exacerbations on health care cost and resource utilization in chronic obstructive pulmonary disease patients with chronic bronchitis from a predominantly Medicare population. Int J Chron Obstruct Pulmon Dis. 2012;7:757-764. doi: 10.2147/COPD.S36997.
6. Morbidity & mortality: 2012 chartbook on cardiovascular, lung, and blood diseases. National Heart, Lung, and Blood Institute website. nhlbi.nih.gov/files/docs/research/2012_ChartBook.pdf. Published February 2012. Accessed January 15, 2017.
7. Blanchette CM, Dalal AA, Mapel D. Changes in COPD demographics and costs over 20 years. J Med Econ. 2012;15(6):1176-1182. doi: 10.3111/13696998.2012.713880.
8. Blanchette CM, Gross NJ, Altman P. Rising costs of COPD and the potential for maintenance therapy to slow the trend. Am Health Drug Benefits. 2014;7(2):98-106.
9. Perera PN, Armstrong EP, Sherrill DL, Skrepnek GH. Acute exacerbations of COPD in the United States: inpatient burden and predictors of costs and mortality. COPD. 2012;9(2):131-141. doi: 10.3109/15412555.2011.650239.
10. Yu AP, Yang H, Wu EQ, Setyawan J, Mocarski M, Blum S. Incremental third-party costs associated with COPD exacerbations: a retrospective claims analysis. J Med Econ. 2011;14(3):315-323. doi: 10.3111/13696998.2011.576295.
11. Abudagga A, Sun SX, Tan H, Solem CT. Exacerbations among chronic bronchitis patients treated with maintenance medications from a US managed care population: an administrative claims data analysis. Int J Chron Obstruct Pulmon Dis. 2013;8:175-185. doi: 10.2147/COPD.S40437.
12. Hurst JR, Vestbo J, Anzueto A, et al; Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints (ECLIPSE) Investigators. Susceptibility to exacerbation in chronic obstructive pulmonary disease. N Engl J Med. 2010;363(12):1128-1138. doi: 10.1056/NEJMoa0909883.
13. Niewoehner DE, Lokhnygina Y, Rice K, et al. Risk indexes for exacerbations and hospitalizations due to COPD. Chest. 2007;131(1):20-28. doi: 10.1378/chest.06-1316.
14. Aaron SD, Donaldson GC, Whitmore GA, Hurst JR, Ramsay T, Wedzicha JA. Time course and pattern of COPD exacerbation onset. Thorax. 2012;67(3):238-243. doi: 10.1136/thoraxjnl-2011-200768.
15. Miravitlles M, Guerrero T, Mayordomo C, Sánchez-Agudo L, Nicolau F, Segú JL. Factors associated with increased risk of exacerbation and hospital admission in a cohort of ambulatory COPD patients: a multiple logistic regression analysis. the EOLO Study Group. Respiration. 2000;67(5):495-501. doi: 10.1159/000067462.
16. Garcia-Aymerich J, Monsó E, Marrades RM, et al; EFRAM Investigators. Risk factors for hospitalization for a chronic obstructive pulmonary disease exacerbation. EFRAM study. Am J Respir Crit Care Med. 2001;164(6):1002-1007. doi: 10.1164/ajrccm.164.6.2006012.
17. Lee SD, Huang MS, Kang J, et al; Investigators of the Predictive Ability of CAT in Acute Exacerbations of COPD (PACE) Study. The COPD assessment test (CAT) assists prediction of COPD exacerbations in high-risk patients. Respir Med. 2014;108(4):600-608. doi: 10.1016/j.rmed.2013.12.014.
18. Bertens LC, Reitsma JB, Moons KG, et al. Development and validation of a model to predict the risk of exacerbations in chronic obstructive pulmonary disease. Int J Chron Obstruct Pulmon Dis. 2013;8:493-499. doi: 10.2147/COPD.S49609.
19. Marin JM, Carrizo SJ, Casanova C, et al. Prediction of risk of COPD exacerbations by the BODE index. Respir Med. 2009;103(3):373-378. doi: 10.1016/j.rmed.2008.10.004.
20. Müllerova H, Maselli DJ, Locantore N, et al. Hospitalized exacerbations of COPD: risk factors and outcomes in the ECLIPSE cohort. Chest. 2015;147(4):999-1007. doi: 10.1378/chest.14-0655.
21. Mapel DW, Schum M, Lydick E, Marton JP. A new method for examining the cost savings of reducing COPD exacerbations. Pharmacoeconomics. 2010;28(9):733-749. doi: 10.2165/11535600-000000000-00000.
22. Montserrat-Capdevila J, Godoy P, Marsal JR, Barbé F, Galván L. Risk factors for exacerbation in chronic obstructive pulmonary disease: a prospective study. Int J Tuberc Lung Dis. 2016;20(3):389-395. doi: 10.5588/ijtld.15.0441.
23. Stanford RH, Nag A, Mapel DW, et al. Validation of a new risk measure for chronic obstructive pulmonary disease exacerbation using health insurance claims data. Ann Am Thorac Soc. 2016;13(7):1067-1075. doi: 10.1513/AnnalsATS.201508-493OC.
24. Summary table of measures, product lines and changes. National Committee for Quality Assurance website. ncqa.org/Portals/0/HEDISQM/HEDIS2016/HEDIS%202016%20List%20of%20Measures.pdf. Published June 16, 2016. Accessed January 9, 2018.
25. Area Health Resources Files (ARF) 2011—2012. Health Resources & Services Administration website. datawarehouse.hrsa.gov/Topics/Ahrf.aspx. Published September 30, 2016. Accessed January 9, 2018.
26. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8-27.
27. Austin PC, Tu JV. Bootstrap methods for developing predictive models. Am Stat. 2004;58(2):131-137.
28. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29-36. doi: 10.1148/radiology.143.1.7063747.
29. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172;discussion 207-212. doi: 10.1002/sim.2929.