James M. Naessens, ScD

With the advent of healthcare payment reform, identifying high-risk populations has become more important to providers. Existing risk-prediction models often focus on chronic conditions. This study sought to better understand other factors to improve identification of the highest risk population. 

A retrospective cohort study of a paneled primary care population utilizing 2010 data to calibrate a risk prediction model of hospital and emergency department (ED) use in 2011. 

Data were randomly split into development and validation data sets. We compared the enhanced model containing the additional risk predictors with the Minnesota medical tiering model. The study was conducted in the primary care practice of an integrated delivery system at an academic medical center in Rochester, Minnesota. The study focus was primary care medical home patients in 2010 and 2011 (n = 84,752), with the primary outcome of subsequent hospitalization or ED visit. A total of 42,384 individuals derived the enhanced risk-prediction model and 42,368 individuals validated the model. Predictors included Adjusted Clinical Groups—based Minnesota medical tiering, patient demographics, insurance status, and prior year healthcare utilization. Additional variables included specific mental and medical conditions, use of high-risk medications, and body mass index. 

The area under the curve in the enhanced model was 0.705 (95% CI, 0.698-0.712) compared with 0.662 (95% CI, 0.656-0.669) in the Minnesota medical tiering-only model. New high-risk patients in the enhanced model were more likely to have lack of health insurance, presence of Medicaid, diagnosed depression, and prior ED utilization. 

An enhanced model including additional healthcare-related factors improved the prediction of risk of hospitalization or ED visit.

		Many risk models that predict hospitalization or emergency department (ED) use are based on comorbid health burden. 

		We investigated other potential predictors, including previous utilization, and found an association of higher hospitalization and ED use with the following: Minnesota medical tiering, 2 prior ED visits, warfarin use, no insurance or Medicaid, being aged over 70 years, and obesity stage 3.

		These findings help to develop new models for determining patients at risk for adverse health outcomes. They focus on potential future areas of clinical practice and research on social determinants of health.

Changing the payment incentives to improve the organization of care delivery, including accountable care organizations, has led to a reorganization of delivery of care focusing on patient-centered medical homes.

 Many patient-centered medical homes stratify their populations and try to align the level of care with the needs of their patients. Providers manage the most complex patients with increased resources to avoid adverse outcomes requiring high-cost care in acute settings.

Two key questions need to be answered: Who is likely to have utilization, and what can be done for these patients? These medically complex patients could be better managed through care coordination programs that are integrated within the delivery system, such as Guided Care, which incorporates specially trained nurses to supplement primary care in a comprehensive care model that provides coordinated, patient-centered primary care for patients with complex needs.

 Different methods for identifying these patients have been developed, including provider identification and referral.

 A hybrid system that utilizes provider selection and an automated risk stratification model may be ideal. Most of the currently available public and proprietary risk models and instruments to identify the population for inclusion in more intensive management programs are based on number and combination of comorbidities. These instruments include self-administered tools such as the Predicting Risk of Admission (PRA) survey, which incorporates self-rated health and comorbid conditions as well as prior healthcare utilization.

 Another model utilizing medical record data is the Elder Risk Assessment (ERA) index, which determines risk of hospitalization based upon increasing age and previous hospitalization as well as comorbid health conditions.

A third risk model, Adjusted Clinical Groups (ACG), uses medical care claims data and stratifies patients based on age and existing chronic conditions.

 The state of Minnesota pays medical homes a care management fee based on Minnesota medical tiering. Minnesota medical tiering is a system of classifying and counting comorbidities originally based on collapsing chronic diseases from ACGs into a body system count.

 This system can also be manually applied without the proprietary software. Our clinical practice uses Minnesota medical tiering to identify appropriate patients for care management. Our prior work comparing these instruments, including Minnesota medical tiering, indicates that these instruments are similar in terms of predicting the risk of hospitalization or emergency department (ED) visit; however, the ACG performed slightly better while the Charlson Comorbidity Index performed worse.

 Although the administrative data-based risk identification methods have been validated and can predict hospitalization with C statistics of 0.67-0.73,

 other potential predictors may improve current risk prediction models and could provide information useful in developing tailored interventions to manage a patient’s condition.

Our clinical experience indicates that coordinating the management of multiple chronic conditions is important in reducing rehospitalizations among geriatric patients with multimorbidity, but other factors may also be important in predicting and managing an individual’s risk.

 In our practice, many patients predicted to be high risk are excluded from care management interventions because they have concurrent mental health or social conditions that care coordinators are unprepared to address. Body mass index (BMI), medication use,

 living environment, English as a second language, previous healthcare use including specialty visits, insurance, and mental health status

 might all be important factors. Patients with those issues may need a different approach, including more support from social services, pharmacy, and psychology.

Obesity is known to influence utilization, but it is widely underdiagnosed and so may be missed in many comorbidity counts.

 The use of anticoagulation therapy, pain, and diabetic medications can lead to an increased need for unplanned acute care and may be amenable to pharmacy consultation.

 Patients living on their own or in dysfunctional families may not be able to cope with increased treatment burdens and may benefit from social services.

 Similarly, those who need language interpretation services may need extra assistance. Patients with a high number of specialty visits may have uncoordinated care and may need help in coordinating their care. Lack of insurance and lower socioeconomic status are associated with increased rates of hospitalization.

 Patients experiencing mental health issues such as depression can also have higher readmission rates.

 Providers and systems with established comorbid health models should understand the potential roles of these additional predictors for hospitalization. 

It is unclear how well these additional factors would predict hospitalization when added to an already established clinical model based on administrative data. To answer this question, we performed a retrospective cohort study comparing a standard risk stratification model of Minnesota medical tiering with an enhanced model containing additional predictors of demographics, selected use of medications, mental health conditions, other specific chronic medical conditions, prior healthcare utilization, and insurance status. We sought to understand how well the enhanced model predicts hospitalization and ED use in a subsequent year.

We constructed a retrospective cohort study to develop and validate a clinical prediction model for hospitalization. The study was approved by the Mayo Clinic Institutional Review Board.

The study was conducted at an academic medical center in Rochester, Minnesota. The population came from the primary care medical practice, Employee and Community Health (ECH), which includes 4 primary care clinics in Dodge and Olmsted Counties, Minnesota. Primary care is provided by family medicine, primary care internal medicine, and community pediatric and adolescent medicine.

Patients 18 years or older who were assigned to a primary care provider within any of the primary care clinics for all 12 months in 2010 (base year) and throughout 12 months or until death in 2011 (assessment year) were included in the analysis. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law. Of the 90,411 patients within the ECH program, as of December 31, 2010, 84,752 (93.7%) gave consent for medical record review.

All information for enrolled ECH patients was electronically abstracted from the electronic medical record (EMR) and administrative databases for 2010 and 2011.

We modeled a combined binary outcome of hospitalization and/or ED visit (hereafter: hosp/ED) at St. Mary’s Hospital or Rochester Methodist Hospital in the assessment year. Hospitalization was defined as any elective or emergent overnight. To avoid double counting, ED visits exclude those resulting in a hospitalization.

Demographic variables—age, gender, marital status, and socioeconomic status&mdash;were collected at the end of the base year. Age was grouped into 6 categories: 18 to 29 years, 30 to 39 years, 40 to 49 years, 50 to 59 years, 60 to 69 years, and 70 years or older. Socioeconomic status was determined by insurance type (Medicaid or no insurance versus other insurance type).

As Minnesota medical tiering is the basis for care coordination payments for patient-centered medical homes in Minnesota, we used it as the basis for our risk model development. Medical conditions are based on diagnosis codes billed by clinical providers.

 Minnesota medical tiering has 5 categories from 0 with no conditions to tier 4 with more than 10 conditions. In our care management program, high-risk patients possess Minnesota medical-tiering scores of 3 or 4. 

Candidate predictors for identifying high-risk patients included primary speaking language not being English; high-risk medications (warfarin, insulin, narcotics); and previous healthcare utilization in specific settings (ED visits, hospitalization, specialty provider visits). Medications were based on the presence of outpatient pharmacy orders in 2010 written by any healthcare providers within the system. Medication use was determined from the EMR, not from filled prescriptions. The medications included warfarin, narcotics and insulin. Non—English speaking status was assigned to those patients needing an interpreter, which was documented within the electronic health record. We included all diagnosis codes (

International Classification of Diseases, Ninth Revision, Clinical Modification

]) from hospitalizations, ED visits, and primary and specialty care visits to identify comorbid mental health and medical conditions, based on a published method.

BMI—recorded as weight (kilograms) / height (meters)

—was also included as a potential predictor. BMI was assessed as being the most recent value in the record within 2 years prior to January 1, 2011. Specific BMI categories followed standard World Health Organization criteria: underweight (BMI <18.5); normal weight (BMI 18.5-24.9); overweight (BMI 25-29.9); stage 1 obesity (BMI 30-34.9); stage 2 obesity (BMI 35-39.9); and stage 3 obesity (BMI >40).

We developed the enhanced model to identify those patients at high risk of hosp/ED within the next year. The 2 models compared in the analysis consisted of Minnesota medical tiering and the enhanced model, which incorporated additional candidate factors. Data were randomly divided into development (50%) and validation cohorts.

We derived the final enhanced model from a stepwise logistic regression followed by a sensitivity analysis seeking stability in the predictors determined by the severity of the outcome variable. Predictors in the saturated model consisted of 72 main factors as well as all possible 1-way and 2-way interaction terms. We determined stability of the variables by changing the number of ED visits or inpatient (IP) visits in the outcome variable. We performed the saturated stepwise model while incrementally increasing the number of ED visits or IP visits. The final set of variables for each change in outcome variable was captured. We then considered a count of how many times a variable had a 

 value ≤.05 in the model. Finally, we included any variable that remained significant in 4 or more of the 11 models considered.

The stepwise regression was repeated 100 times and each repetition considered the variables in random order to determine if order affected the results of the stepwise algorithm. Each iteration produced the same model. Effects were entered into and removed from the model so that each forward selection step could be followed by a backward elimination step. The algorithm stopped when no further effects could be added to the model or if the current model was identical to a previous model.

Comparisons between prediction models focused on predictability and precision. We used C statistics (area under the receiver operating characteristic [ROC] curve [AUC]) as our measure of predictability. For assessment of precision, we compared the sets of high-risk patients found within the tier 3 and 4 groups in Minnesota medical tiering with an equal number of high-scoring patients from the enhanced model. Statistical analysis was performed using SAS version 9.3 (SAS Institute, Cary, North Carolina).

A total of 42,384 and 42,368 patients were included in the development and validation samples, respectively. The development and validation data sets were similar. There were no differences in sex, age decile, BMI, marital status, insurance status, prior healthcare utilization, comorbid health status, or Minnesota medical tiering (

). In addition, the 2 samples had similar rates of hospitalization and ED visits with 18.2% of the development sample and 17.6% of the validation sample having a hospitalization or ED visit in the subsequent year.

Those patients who belonged to the highest category of Minnesota medical tiering (4) had the greatest odds of incurring any hospitalization or ED visit with an odds ratio of 18.2 (95% CI, 15.6-21.24) compared with the lowest tier category (0). Next most important was prior utilization of 2 or more ED visits with an odds ratio of 7.13 (95% CI, 6.34-8.01) for predicting hosp/ED. Similarly, 2 or more previous hospital stays had an odds ratio of 4.28 (95% CI, 3.90-4.69) for predicting hosp/ED. Mental health conditions, including substance abuse, personality disorder or dementia, each had odds near 3 for predicting hosp/ED. Specifically, depression had an odds ratio of 2.92 (95% CI, 2.71-3.15). Among other medical conditions, cardiac disease with heart failure and pulmonary heart disease had odds ratios over 5. All specific conditions with odds ratios over 2 are reported in Table 1. All specific conditions are listed in 

. Patients in each of the medication categories had odds ratios greater than 2, as did those utilizing 3 or more specialist visits in the prior year. Patients with a language interpreter did not have elevated risk of hosp/ED (Table 2).

The multivariable enhanced tiering model is presented in 

. The final model included 30 main effects (representing 16 factors) and 4 2-way interaction terms (more detailed results are available from the authors).

Of notable mention, being in the highest medical tier remains the strongest predictor for hosp/ED after the addition of other factors with an odds ratio of 3.50 (95% CI, 2.78-4.42). More than 1 previous ED visit also resulted in an increased odds ratio of 3.17 (95% CI, 2.78-3.60). Several significant interaction terms are noted in Table 3. In the validation cohort, the C statistic for the enhanced multivariable model was 0.711 (95% CI, 0.704-0.717) compared with a C statistic for the Minnesota medical tiering alone model of 0.667 (95% CI 0.660-0.673). These C statistics were actually slightly higher in the validation cohort than in the development cohort (enhanced model: 0.705 [95% CI, 0.698-0.712]; Minnesota medical tiering: 0.662 [95% CI, 0.656-0.669]). The differences between the ability of the enhanced model to predict hospitalization and ED use and Minnesota medical tiering are displayed in the ROC curves found in the 

Many different patients were identified as being high-risk in the enhanced model as compared with those in the original Minnesota medical tiering model (tier 3 or 4). 

 presents the characteristics of high-risk patients identified with both models as well as those unique to the 2 approaches. We found that patients identified as high-risk in the enhanced model were much more likely to experience hospital utilization in 2011 than those in the Minnesota medical tiering-only model (

 <.001). Patients identified as high-risk based only on the enhanced model, compared with those identified only through Minnesota medical tiering, were of younger age, had higher previous healthcare utilization, and had more frequent mental health conditions. Many of the high-risk enhanced model patients were aged less than 40 years compared with only 11% of patients being aged less than 40 years in the Minnesota medical tiering model.

In this retrospective cohort of patients receiving care within the ECH, several variables were important in predicting hosp/ED in addition to the number of chronic comorbidities currently used in Minnesota medical tiering and our care coordination identification. We found that age, BMI, previous healthcare utilization, mental health conditions, anemia, heart failure, epilepsy, hyperlipidemia, warfarin use, and narcotic use all simultaneously contributed to the prediction of hospitalization and ED visits in the next year. This enhanced model had an AUC of 0.711 (95% CI, 0.704-0.717) for predicting hosp/ED, as compared with 0.667 (95% CI, 0.660-0.673) found in the model utilizing only Minnesota medical tiering. Previous work comparing different risk stratification methods for hospitalization determined slight superiority of ACG compared with the ERA, Hierarchical Condition Categories, Minnesota medical tiering, and chronic care counting instruments.

 The enhanced model takes into consideration previous hospitalization, specific high-risk illnesses, mental health conditions, and high-risk medication use (eg, warfarin, narcotics) that are not universally accounted for in other models. In comparing high-risk groups of the same size from both models and excluding patients identified in both models, 47% of the patients in the enhanced model experienced hosp/ED compared with 32.5% of patients from the Minnesota medical tiering model. The enhanced model accounts for factors in a younger population (eg, mental health and previous utilization) that are not detected in the Minnesota medical tiering model.

Previous hospital and ED utilization served as strong predictors for hosp/ED in the following year. Specifically, 2 or more prior ED visits presented an odds ratio (OR) close to 3 while 1 prior hospital stay resulted in an OR of 1.79 (95% CI, 1.66-1.93). Together, hospitalization and ED visits often indicate a medically complex individual or an individual with severe illness. Many risk stratification models have utilized previous healthcare usage as a predictor for future healthcare use. The PRA incorporates previous hospitalization within the past year as a factor predictive of future hospitalization.

 Similarly, the ERA also includes previous hospital days to calculate a risk score for future hospitalization.

 Previous ED visits as well as previous hospitalization have been associated with increased hospital admission in a multi-centered VA study of 1378 patients designed to identify factors of hospitalization.

Articles

An enhanced risk model incorporating medication use, prior healthcare utilization, and mental health with comorbid health conditions predicts healthcare utilization better than health conditions alone.  

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Published literature should be more critically evaluated when the chances of safety, outcome, or cost implications are high

		Abstracts can use overstated results or terms to drive readership and citation rates.

		Since the abstract may be the only part of a paper that administrators read, errors in abstracts can have significant real-world impacts on decision making.

		Errors in interpretation of results can also have significant real-world impacts on decision making that can ripple across the entire spectrum of healthcare management.

 that compares different risk stratification methods is enlightening, although the authors have overstated their results and a discrepancy between the abstract and their results both contribute to the possibility that their study could be misinterpreted in costly ways.

It is good practice to include CIs on all point estimates, and CIs are at least available for the main results presented in 

 (on following page). But the authors fail to interpret those values, and as a result conclude—incorrectly&mdash; that the Adjusted Clinical Group (ACG) model is the best. Given the information available in the paper, it is correct to say that the results are consistent with there being no difference between ACG and Minnesota Tiering (MN). In fact, the paper’s figures (which lack CIs, unfortunately) and C statistic values suggest that any differences in outcomes for all models might be statistically distinct at times but are practically trivial. It is also questionable whether the C statistic is even appropriate for comparing models with different drivers, and whether it accurately portrays the receiver operating characteristic curve upon which it rests (it can’t).

The specific error in the abstract states that the C statistic for the ACG model for predicting the highest 10% of cost users is 0.81, but Table 2 shows that the correct value is 0.76 (95% CI, 0.75-0.76). Since this overlaps with the results for the MN model (0.74; 95% CI, 0.74-0.75), the ACG model is not clearly “superior to the others.” If the authors meant to refer to readmissions, again the overlap in CIs between ACG and MN show that they are essentially equivalent models in this outcome as well. The term “superior” is a value-laden word that can be easily overinterpreted without a close read of the results; its use here presents both clerical and conceptual errors.

It may well be that the ACG model is superior to these other models, but this paper did not demonstrate that—in fact, this study’s results actually support the idea that all of these models are practically equivalent. Academics might call that conclusion a “negative result” and decide not to publish because of the excess difficulty of getting it through review. But as an industry professional, I would consider “negative results” like these&mdash;when properly interpreted&mdash; “fiscally important results.”

 is read by thousands of on-the-ground healthcare industry workers, and time-constrained administrators often take abstracts at face value, having no time to evaluate the results themselves. In fact, this paper made its rounds among our administrators when it was published, and may (or may not) have contributed to our subsequent decision to purchase one of these models. Purchasing risk models is expensive, and internal switching, implementation, and training costs multiply that well beyond the cost of the model. I would ask that editors and reviewers make absolutely sure that authors’ conclusions are warranted by their results before publication—improper interpretation can be as damaging and/or costly as clerical errors when subjected to the decision-making hurricane that is today’s healthcare industry.

The letter “Simple Errors in Interpretation and Publication Can Be Costly,” a response to the article “Risk-Stratification Methods for Identifying Patients for Care Coordination,” is primarily based on 1 line in the abstract. The author of the letter did identify a typographical error in the abstract which included the C statistic for readmissions instead of that for high-cost users (we have submitted an erratam to correct the error). Although we agree that the word “superior” could have been softened, the ACG did perform better than other models in predicting healthcare utilization. For example, as can be seen in Table 2 (below) of the original paper, for predicting the top 10% high-cost users, the ACG had non-overlapping CIs with all models except for MN Tiering, which is based on the ACG.

We believe the abstract was clear. Unfortunately, as abstracts are limited with regards to space, we would hope the reader would draw their impressions from the entire paper.â€‹ The conclusions of the paper stated that although the ACG was generally better at predicting utilization, all models had good concordance, suggesting that choosing any model would be more beneficial than none. All models, excluding ACG and MN Tiering, are free and publically available.

We agree with the correspondent that receiver operating characteristic curves should not be the only basis of comparison between models. In fact, the analysis did not rely strictly on C statistics, but also focused on calibration, particularly regarding the accuracy of identifying patients at the high end of each of the outcome distributions as displayed in Table 3 and Figure 2 (on following page).

 Dwight Barry, PhD, Group Health Cooperative, 320 Westlake Ave N, Ste 100, Seattle, WA 98109. E-mail: barry.d@ ghc.org.

The authors (LRS, PYT, NDS, RJS, MEB, DMF, JMN) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of the article.

: Concept and design (LRS, PYT, NDS, RJS, MEB, DMF, JMN); acquisition of data (LRS, JMN); analysis and interpretation of data (LRS, NDS, RJS, MEB, DMF, JMN); drafting of the manuscript (LRS, PYT, MEB, JMN); critical revision of the manuscript for important intellectual content (LRS, PYT, NDS, RJS, MEB, JMN); statistical analysis (LRS, JMN); obtaining funding (JMN); and administrative, technical, or logistic support (DMF).

Lindsey R. Haas, MPH, Health Care Policy and Research, Mayo Clinic, 200 First St Southwest, Rochester, MN 55905. E-mail: haas.lindsey@mayo.edu. 1. Haas LR, Takahashi PY, Shah ND, et al. Risk-stratification methods for identifying patients for care coordination. Am J Manag Care. 2013;19(9):725-732.

2. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935.

3. Hand DJ. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn. 2009;77:103-123.

4. Trouble at the lab. The Economist. 2013;409(8858):26-30. 

A recent AJMC study contained overstatements and small but importantly placed errors that have the potential to cause unwarranted on-the-ground cost problems.

Simple Errors in Interpretation and Publication Can Be Costly

Care coordination is a key component of the patient-centered medical home. However, the mechanism for identifying primary care patients who may benefit the most from this model of care is unclear.

To evaluate the performance of several risk-adjustment/stratification instruments in predicting healthcare utilization.

 All adults empaneled in 2009 and 2010 (n = 83,187) in a primary care practice were studied. We evaluated 6 models: Adjusted Clinical Groups (ACGs), Hierarchical Condition Categories (HCCs), Elder Risk Assessment, Chronic Comorbidity Count, Charlson Comorbidity Index, and Minnesota Health Care Home Tiering. A seventh model combining Minnesota Tiering with ERA score was also assessed. Logistic regression models using demographic characteristics and diagnoses from 2009 were used to predict healthcare utilization and costs for 2010 with binary outcomes (emergency department [ED] visits, hospitalizations, 30-day readmissions, and highcost users in the top 10%), using the C statistic and goodness of fit among the top decile.

The ACG model outperformed the others in predicting hospitalizations with a C statistic range of 0.67 (CMS-HCC) to 0.73. In predicting ED visits, the C statistic ranged from 0.58 (CMSHCC) to 0.67 (ACG). When predicting the top 10% highest cost users, the performance of the ACG model was good (area under the curve = 0.76) and superior to the others.

 Although ACG models generally performed better in predicting utilization, use of any of these models will help practices implement care coordination more efficiently.

For care coordination to be used effectively, it is important to identify which patients are likely to benefit from it. Little evidence exists as to which risk-stratification method performs better in predicting hospitalizations, emergency department visits, 30-day readmissions, and high-cost users.

		There appeared to be good concordance among 6 different risk screening instruments in predicting hospitalization.

		Adjusted Clinical Group classification was generally better than other models in predicting healthcare utilization.n

		Focusing care coordination efforts within the medical home on patients likely to benefit most requires appropriate identification of the highest risk, highest utilizing patients.

With rising prevalence of chronic disease, patient-centered medical homes (PCMHs) were created to deliver higherquality, more cost-effective primary care in the United States.

 A key component of PCMH is the implementation of care coordination, especially if targeted to the right people.

 While some states have developed risk-adjusted care coordination payments,  methods for effective identification of individuals likely to benefit from care coordination are not clear.

Patients with multiple chronic conditions may see many providers, often in an uncoordinated manner, adding to their increased risk of high healthcare utilization through test duplication, edication management conflicts, and medical errors.

These patients are also at increased risk for hospital admissions and emergency department (ED) visits. Primary care doctors may not routinely have the time or resources to properly manage these complex patients who have multiple chronic health problems. Thus, care  coordination for the right patients could decrease unnecessary care and prevent adverse outcomes.

Most existing PCMH programs identify care coordination patients through physician referrals.

 However, this method may not identify all potential beneficiaries. Care coordination should be focused on patients who will benefit most, maximizing the impact on both quality and costs. Risk-stratification models can be efficient tools for both providers and health plans to screen populations and select individuals for care management programs who are at risk of future hospitalization and functional decline.

 Several methods using administrative data have been described to help with the identification of high-risk  patients.

 However, it remains unclear which method will perform best in identifying patients in need of care coordination. To answer this important and fundamental question, we compared the performance of 6 common risk-adjustment methods in predicting hospitalizations, ED visits, 30-day readmissions, and high expenditures. This study extends the existing literature on patient selection for care coordination with results from direct comparison of the following models: Adjusted Clinical Groups (ACGs), Hierarchical Condition Categories (HCCs), Elder Risk Assessment (ERA), Chronic Comorbidity Count (CCC), Charlson Comorbidity Index, and Minnesota Health Care Home Tiering (MN Tiering). The identification of an optimal method may guide implementation of care coordination programs within PCMHs. Furthermore, identifying these predictable  high-utilization cases may lead to other effective interventions.

A retrospective cohort analysis compared multiple methods to identify high-risk patients. These methods were applied to all primary care patients 18 years or older empaneled to the Employee and Community Health (ECH) practice (family medicine, primary care internal medicine, and community pediatric and adolescent medicine) at Mayo Clinic, Rochester, Minnesota, the community-focused primary care arm of a large integrated multispecialty group practice.

To be a patient of ECH, one must be a Mayo Clinic employee or live in the community, identify an ECH primary care provider, and be enrolled within his/her panel. Participants assigned to a primary care provider for all 12 months in 2009 (base year) and throughout 12 months or until death in 2010 (assessment year) based on the electronic medical record were included in the analysis. The organized development of care coordination within ECH occurred during 2011; hence, usual care (encounter driven) was provided to all patients during the study. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law.

All information was electronically abstracted from the electronic medical record and administrative databases within Mayo Clinic’s health records system. Mayo Clinic maintains all electronic medical record information within 1 system, including inpatient and outpatient visit information.

Demographic variables collected in the base year included age, sex, marital status, and insurance status. Age was grouped into 4 categories of 18 to 44 years, 45 to 64 years, 65 to 84 years, and more than 85 years. Insurance status was also grouped into 4 categories: public (Medicare or Medicaid), private, no insurance, and unknown. Diagnosis codes (

International Classification of Diseases, Ninth Revision, Clinical Modification 

) for each patient encounter, as well as utilization and cost information, were extracted from institutional billing data for year 2009. Number of hospital days was also retrieved from 2008 to calculate ERA scores. We included all diagnosis codes from hospitalizations, ED visits, and primary and specialty care evaluation and management visits.

The ACG is a commonly used classification system based on administrative diagnosis data and developed by Johns Hopkins to measure morbidity.

 The ACG methodology was developed to predict the utilization of medical resources using the presence or absence of specific diagnoses from both inpatient and outpatient services for a specified period of time, along with age and sex, to classify each person into 1 of 93 discrete ACG categories with similar expected utilization patterns. Adjusted Clinical Groups can be used to improve accuracy and fairness in forecasting healthcare utilization

 and have been found to predict inpatient hospitalizations as well as or better than other case-mix tools in many health systems.

 The MN Tiering model is based on a product of ACGs, Major Extended Diagnostic Groups.

 MN Tiering is currently being used by the State of Minnesota to determine management fees for care coordination among medical home plans. The purpose of MN Tiering is to group patients into “complexity tiers” based on the number of major condition categories from which they suffer.3 The total sum of conditions is grouped into the following 5 patient complexity levels: low (tier 0): 0  conditions; basic (tier 1): 1 to 3 conditions; intermediate (tier 2): 4 to 6 conditions; extended (tier 3): 7 to 9 conditions; and complex (tier 4): 10 or more conditions.

In 2004, the Centers for Medicare & Medicaid Services (CMS) implemented HCCs to adjust Medicare capitation payments for health expenditure risk of Medicare Advantage Plan (health maintenance organization) enrollees.

 diagnosis codes and demographic data for each patient are aggregated into 70 condition categories that contribute to a single risk score.17 Several studies among Medicare patients have provided evidence that HCC scores for risk adjustment can be effective at predicting hospitalizations.

 Elder Risk Assessment Index. The ERA Index was developed to identify patients at risk for hospitalization and ED visits in adults 60 years or older.

The ERA Index incorporates a weighted score of age, sex, number of hospital days in the prior 2 years, and marital status, as well as selected medical conditions (diabetes, coronary artery disease, congestive heart failure, stroke, chronic obstructive pulmonary disease,and dementia).

The minimum score on the index is —1 and the maximum score possible is 34.

The CCC method was derived by Naessens and colleagues

 from a modification of the method of Hwang and colleagues,

 based on the publicly available Agency for Healthcare Research and Quality’s Clinical Classification Software. The CCC method is more comprehensive than most comorbidity counts. The total sum of chronic conditions was grouped into 6 categories: 0, 1, 2, 3, 4, and 5 or more. Comorbidity counts have been shown to be associated with high annual costs as well as persistence in high costs.

 The Charlson Comorbidity Index, originally derived to classify comorbidities affecting 1-year mortality in cancer patients, sums weights for 17 specific conditions.23 In our study, the sum of Charlson Comorbidity Index counts was used to predict future outcomes. The performance of the Charlson Comorbidity Index in predicting poor outcomes has been assessed in various large populations, and its validity as a prognostic measure of outcomes has been consistently demonstrated.

 A care coordination patient enrollment process was developed in our primary care practice based on combining MN Tiering and the ERA score as a hybrid model. Patients eligible for enrollment in care coordination included all patients in MN tier 4 (individuals with more than 10 comorbid conditions) and those in MN tier 3 with ERA scores greater than 10. Although all of the previously mentioned risk-stratification instruments have been validated, it remains unclear how the hybrid method of combining MN Tiering and ERA score will compare with them.

Several outcome measures could be used as surrogates for patients who may benefit from care coordination. We modeled 4 binary outcomes reflecting high utilization within the 12-month prediction period of 2010: (1) any inpatient hospitalization, (2) ED visits not resulting in hospitalizations, (3) any readmission within 30 days of an initial hospitalization, and (4) being a high-cost user. A high-cost user wasdefined using the threshold of the top 10% users in 2010 (>$7457). We chose to focus on the top 10% of users because they consume almost 70% of total costs, but are less influenced by catastrophic conditions and potentially more amenable to the effects of care coordination than patients at a higher threshold. These outcomes are utilization measures believed to be influenced by effective care coordination. 

ED visits resulting in a hospitalization were recorded as a single inpatient hospitalization event. Total costs reflect a provider perspective and were based on standardized inflation-adjusted cost estimates for each service or procedure provided within our healthcare system (not including outpatient pharmacy services) in 2010 constant Medicare dollars.

We applied logistic regression models to compare the 6 risk-stratification methods to determine which method best predicted the dichotomous outcomes in the subsequent year. Model performance was based on explanatory power and goodness of fit. Explanatory power was assessed by using the C statistic with 95% confidence intervals to predict (1) hospitalizations, ED visits, 30-day readmissions, and high-cost users and (2) the ability of each model to identify individuals with the outcomes of interest in the highest and lowest predicted deciles. The C statistic is a measure of model discrimination and is equivalent to the area under the receiver operating characteristic curve.

To assess goodness of fit, we compared the observed and predicted hospitalizations, ED visits, readmissions, and high-cost users in the lowest and highest deciles on the basis of predicted probabilities.

To address calibration, we further focused our assessment on those patients at the highest end of each risk score to identify which patients need care coordination. Since the hybrid model was used clinically in 2011, we used the number of patients considered for PCMH enrollment as a target number to establish the threshold for potential care coordination (based on the highest estimated probability of hospitalizations). We then determined (1) how much overlap in identification occurred in each of the 7 methods and (2) how much of the total utilization the 6 other approaches would have identified. Analyses were conducted with SAS software, version 9.1 (SAS Institute Inc, Cary, North Carolina). The study was approved by the Mayo Clinic Institutional Review Board. No external funding was obtained.

The study population included 83,187 patients who met inclusion criteria between January 1, 2009, and December 31, 2010. The mean age of the base population was 46.9 years, 54.6% were female, 63.1% had private insurance, and 21.8%

). Table 1 shows the frequency distributions of the demographic characteristics at the end of 2009, as well as the percentage of paneled patients with selected chronic diseases.

Healthcare utilization and resource use for the base and prediction years were similar (Table 1). Approximately 8% of patients had a hospitalization, and 13% percent of the cohort had an ED visit. Compared with the base year 2009, the mean total cost in 2010 was nearly the same, decreasing by 1%. We saw the expected concentration of healthcare services among a relatively small number of individuals (

). Overall, 32.4% of the most expensive 10% of patients in 2009 were also in the top 10% of patients in 2010. Furthermore, our outcomes of interest were correlated, though not perfectly. High cost was associated with each of the utilization measures, with  Pearson correlations ranging from 0.22 for ED visits to 0.72 for hospitalizations. The correlations between utilization measures ranged from 0.12 between ED visits and readmission to 0.35 between hospitalization and readmission.

, the ACG model outperformed the other 5 models in predicting hospitalizations, with a C statistic range of 0.67 (CMS-HCC) to 0.73 (ACG). The ERA score and MN Tiering followed close behind ACG for prediction of hospitalization (C statistic = 0.71). In the models predicting ED visits, the C statistic ranged from 0.58 (CMS-HCC) to 0.67, with the ACG model having the best predictive ability. Further, the ACG model outperformed other models when predicting 30-day readmissions; the C statistic ranged from 0.74 (CMS-HCC) to 0.81 (ACG). When predicting healthcare expenditures for the top 10% high-cost users, the performance of the ACG model was good (C statistic = 0.76) and superior to that of other models. CMS-HCC had the lowest predictive ability for all 4 outcomes. It is important to point out that although ACGs had the best predictive ability, much of the variability in outcomes was unexplained by any model. For each outcome, models with higher C statistics also had higher rates of actual events among the highest deciles. For example, the top decile for ACG models had 27% with a hospitalization and 31% with at least 1 ED visit, whereas the top decile for CMS-HCC models had 25% with a hospitalization and 23% with an ED visit. There was more potential discrimination with ACG.

To further evaluate the 6 methods, we compared the patients in the top decile of predicted probability of having a hospitalization (

). The ERA method tended to overpredict for those in the top decile, whereas the CMSHCC method underpredicted. For the ACG, MN Tiering, and CCC methods, the actual and predicted hospitalizations were nearly equivalent. Using the model based on ACGs, 26.8% of patients in the highest decile were hospitalized; using the MN Tiering model, 25.1% in the highest decile were hospitalized; and using the Charlson Comorbidity Index, only 22.9% in the highest decile were hospitalized. Similar results were seen for the other 3 outcomes (results are available in 

A total of 2347 (2.8%) patients were identified as meriting care coordination based on the hybrid clinical approach (Table 3). At least 40% of these base patients were in the selected top group of patients, irrespective of which risk method was used. Interestingly, our initial clinical implementation actually identified the patients with the highest number of hospitalizations, the highest percentage of patients with any ED visit, and the patients with highest total costs.

We assessed 6 risk instrument methods based on administrative and demographic data. We evaluated the performance of the 6 models against one another to assess the ability to predict future healthcare utilization. We concluded that the ACGs produced a more accurate prediction of future healthcare utilization relative to the other models.

All risk prediction models for hospitalization had fair predictive value, with ACG having the highest overall predictive C statistic at 0.73 and the HCC model having the lowest predictive C statistic at 0.67. In a previous large study, the ACG had an excellent predictive area under the curve of 0.80.29 An earlier study on the HCC showed an area under the curve of 0.638 for predicting hospitalizations among newly enrolled Medicare patients.

MN Tiering, CCC, and the ERA all performed similarly; thus, use of each could be justified. These novel and unique findings indicate to both providers and health plans that any of these risk-stratification models can be used for clinical purposes. The individual risk instruments performed in a similar fashion for ED visits, readmissions, and high-cost users as well as hospitalizations, as might be expected, because these outcomes tend to be correlated. The predictive values of the risk-stratification instruments were slightly lower for ED use, but higher for 30-day readmission and high-cost users. These findings provide important information regarding use of newer risk-stratification tools like MN Tiering, ERA, and CCC. The instruments predicted not only hospitalization but also re-hospitalization and ED visits.

Although all rely on the presence of diagnoses and demographic factors, the 6 risk-screening instruments vary with respect to ease of implementation. Unfortunately, the best performing methods, ACG and MN Tiering, are also the only methods we examined that require software licensing. A form of MN Tiering based on manual classification is available.

CMS-HCC is a software package that can be downloaded from CMS. The algorithms for the ERA Index, CCC, and Charlson Comorbidity Index have all been published and are available, but need to be programmed to be applied for local clinical use.

Identifying which patients are likely to benefit from care coordination is important. We evaluated the performance of 6 risk-screening instruments in predicting healthcare utilization.

Risk-Stratification Methods for Identifying Patients for Care Coordination

 To determine the longitudinal effect on healthcare costs of multiple chronic conditions among adults aged 18 to 64 years.

 Retrospective cohort assessment of working-age employees and their dependents with continuous coverage in a self-funded health plan from January 1, 2004, to December 31, 2007. Data were obtained from health benefit enrollment files and from medical and pharmacy claims.

Individuals were defined as having chronic conditions based on modification of a published method. The mean annual healthcare costs were estimated for individuals with 0, 1, 2, 3, 4, or 5 or more chronic conditions. The probability of persistence in high-cost categories across years was estimated for individuals in each group.

 Overall, 75.3% of working-age adult enrollees had at least 1 chronic condition, 54.3% had multiple chronic conditions, and 16.5% had 5 or more chronic conditions. The cost of healthcare was higher among individuals with more chronic conditions for all ages. The mean medical cost per year for an individual with no chronic conditions was $2137, while that for an individual with 5 or more chronic conditions was $21,183. Enrollees with more chronic conditions had higher persistence in high-cost categories between years and persisted at these high costs for more years.

 While multiple chronic conditions are common in the population 65 years and older, they are also of great concern for the working-age population. Understanding how to effectively manage individuals with multiple chronic conditions is an important challenge. Effective care management focused on managing the patient as opposed to a condition has the potential to significantly affect healthcare costs.

This study looked at the longitudinal effect on healthcare costs of multiple chronic conditions among working-age adults.

Results showed that 75.3% of working-age adult healthcare system enrollees had at least 1 chronic condition and that 16.5% had 5 or more chronic conditions.

The cost of healthcare was higher for individuals with more chronic conditions.

Enrollees with more chronic conditions had increased persistence in high-cost categories and persisted at these high costs for more years.

The best care management for working-age adults may be a stratified approach. Effective care management focused on managing the patient as opposed to a condition has the potential to significantly affect healthcare costs.

Cross-sectional investigations have demonstrated the importance of key chronic conditions on the utilization and cost of healthcare services among older populations.

 Medicare patients with 3 or more chronic conditions were estimated to account for more than 92% of 2002 Medicare expenditures.

 Although Medicare patients have been studied extensively, little is known about the prevalence and longer-term implications of multiple chronic conditions among younger adults. The highest prevalence of chronic conditions is among older persons, while most individuals with chronic conditions are working age and privately insured.

 Healthcare expenditures for the nonelderly are rising.

 The rising prevalence of multimorbidity among working-age patients has increased pharmacy needs and the use of healthcare services.

 The indirect costs of absence, short-term disability, and productivity losses add to healthcare costs.

Understanding the prevalence, persistence, and cost of multiple chronic conditions among working-age adults is essential to improving care and managing healthcare costs. This article examines the prevalence of multiple chronic conditions among a population of employees and their dependents with continuous coverage in a large healthcare delivery system from January 1, 2004, to December 31, 2007, and assesses the effect on persistence of healthcare costs over time.

All 33,324 employees and dependents of the Mayo Clinic in Rochester, Minnesota, aged 18 to 64 years with continuous health benefit coverage from January 1, 2004, to December 31, 2007, were included in the study. To avoid potential bias, enrollees who died during the time frame (n = 46) were included. Almost 75% of 45,288 enrollees aged 18 to 64 years in 2007 were included in the analysis.

Continuous healthcare benefit coverage was determined from benefit enrollment records for the 4-year time frame. Data on health-care utilization and costs were obtained from medical and pharmacy claims for eligible individuals.

 Our operational definition of chronic conditions is a modification of that used by Hwang et al.

 Chronic conditions not classified into chronic disease subgroups according to the method by Hwang et al (using Agency for Healthcare Research and Quality Clinical Classifications Software for 

 [http://www.hcupus. ahrq.gov/toolssoftware/ccs/ccs.jsp]) were then assessed for inclusion by a panel among us (JMN, RJS, and DMF). All

diagnosis codes included for any hospital encounter or physician visit between January 1, 2004, and December 31, 2007, were included to determine the chronic conditions for each enrollee.

 Costs were analyzed from the payer perspective as modified to be based on total payments made by any insurance or by the patient for deductibles, coinsurance, or  copayments. Costs were reported in constant 2007 US dollars, with adjustments over time based on the Consumer Price Index (http://www.gpoaccess.gov/eop/2007/b60.oxs). Costs were examined within age groups, in expense categories, and by the number and type of chronic condition. The study sample was rank ordered by both annual and4-year   total expenses and then categorized into the following 4 percentile groups: (1) the top 5%, (2) the next 6% to 20%, (3) the next 21% to 50%, and (4) the bottom 50%. A transition probability matrix was developed to characterize individual changes in healthcare spending from year to year over 4 years.

Prevalence and Cost of Chronic Conditions

The number and percentage of adults identified with specific chronic conditions, as well as the mean annual cost for all enrollees with that chronic condition, are given in 

). The most frequent chronic conditions included hyperlipidemia (29.4%), hypertension (18.9%), depression (16.6%), allergic rhinitis (14.9%), and osteoarthritis (12.7%). Most of these chronic conditions increased with age, some substantially. When considering both prevalence and total costs per patient for all services, the patients with the following chronic conditions had the largest total expenditures: hyperlipidemia, hypertension, depression, osteoarthritis, conduction disorders, allergic rhinitis, backproblems, obesity, malignant neoplasms, and ovarian, uterine, and reproductive problems.

Prevalence and Cost of Multiple Chronic Conditions Across Age Groups

Overall, 75.3% of working-age adult enrollees had at least 1 chronic condition, 54.3% had multiple chronic conditions, and 16.5% had 5 or more chronic conditions. The percentage of individuals with any chronic condition and the percentage with multiple chronic conditions increase with age among working-age adults (

). A larger proportion of total health expenditures is consumed by a smaller proportion of beneficiaries as the number of chronic conditions increases. The mean annual cost per person increases from $1700 to $2000 per additional chronic condition for enrollees with 0 to 4 chronic conditions across inpatient, pharmacy, and physician services. Compared with those having 4 chronic conditions, enrollees with 5 or more chronic conditions have a mean annual cost per person that is about $10,000 higher than that of enrollees with 4 chronic conditions. Annual costs are highest among young adults for each number of chronic conditions. For the other age groups, total costs for individuals with 5 or more chronic conditions increase with age. As expected, most service types increase among persons with more chronic conditions.

Large proportions of individuals in the highest and lowest cost categories for any year remain in the same cost categories for the next year. The 

shows the probabilities that  individuals remain in the top 5% from year to year by the number of chronic conditions. 

) gives the transition probabilities of remaining in or changing between each percentile category from year to year overall and by the number of chronic conditions. The 5% of enrollees with the highest costs in 2004 accounted for 33.3% of total costs in 2004, while the 50% of enrollees with the lowest costs in 2004 accounted for 10.5% of total costs in 2004. Overall, 26.4% of those in the top 5% in the base year are in the highest-cost category in the next year, 20.9% are in the highest-cost category in the third year, and 19.9% remain in the highest-cost category in the fourth year. However, enrollees with more chronic conditions have higher persistence in high-cost categories than those with fewer chronic conditions and persist at these high costs for more years (Figure). Among individuals with 5 chronic conditions, more than 35% of those in the highest category in the base year remained among the costliest patients in the next year, and approximately 30% were among the costliest patients in years 3 and 4. Meanwhile, among those with 1 chronic condition who started in the costliest category, only 7.6% remained in the costliest category in the next year, while 3.9% and 2.7% were in the most costly category in years 3 and 4, respectively.

As has been reported for older populations, the presence of chronic conditions increases the costs of healthcare for working-age adults, not just for a single year but over longer time frames. Patients with many selected chronic conditions have high mean costs, and those with high costs often have multiple problems. The mean costs increase steadily with the addition of each chronic condition from 0 to 4, with greater cost increases for 5 or more chronic conditions. The number of chronic conditions increases the probability of  persisting in high-cost categories.

Patients with multiple chronic conditions pose a complex set of problems. Among these are disruptions in continuity of care when patients transfer between primary care and specialty providers.

 Adherence to clinical care guidelines across multiple chronic conditions is also problematic, as most guidelines do not apply to multiple chronic conditions.

 Disorganized or uncoordinated care of complex patients is costly. Duplication of tests and services,

 breakdown of communication in the referral process,

  readmissions that could have been prevented with coordinated care,

 and medical errors or forgoing of medical care are costly and potentially fatal.

 reported that up to 88% of the population 65 years and older had 1 or more medical chronic conditions, with up to 78% of total healthcare expenses going toward treating individuals with chronic conditions in 2004, and,Wolff et al

 found that 82% of Medicare-eligible patients had chronic conditions and 65% had multiple chronic conditions in 1999. This is comparable to our finding that 75.3% of adults aged 18 to 64 years have a chronic condition and that these individuals consume 93.1% of total healthcare expenses.  Multiple chronic conditions were seen among 54.3% of our cohort. The differences could be due to age but also may reflect better health among employed individuals with insurance coverage. A steady increase in healthcare costs among individuals with additional chronic conditions was also reported by Wolff et al.

Herein, the highest-cost patients with multiple chronic conditions consistently were in high-cost categories across the 4-year time line. This is in agreement with a 3-year study

 among Medicaid patients that found similar persistence of beneficiaries in high-cost categories.

The generalizability of our results may be limited by the fact that we studied employees and dependents of a single healthcare system in the Midwest. Furthermore, our results are based on a population with continuous insurance coverage for multiple years. The rich employer-based medical, disability, and retirement benefits may have led to higher retention among individuals with multiple chronic conditions covered by our health plan compared with working-age adults covered by similar plans. We found that 75.3% of health plan enrollees who were continuously eligible for 4 years had at least 1 chronic condition, while 37.9% had 3 or more chronic conditions. Paez and colleagues

found that 45.3% of respondents aged 18 to 64 years had at least 1 chronic disease, while 12.0% had 3 or more chronic conditions in the 2005 Medical Expenditure Panel Survey data. Although our study and their study were based on the classification of chronic conditions by Hwang et al,1 there are important differences between the 2 studies in how chronic conditionsare  identified (claims data vs self-report) and in what the time frame of interest was (4-year longitudinal vs cross-sectional). These observations are balanced by the fact that our study was based on 4 years of longitudinal assessment of a non-Medicare population. We were able to study a large cohort among a stable employee with uniform insurance coverage, primarily served by an integrated provider group with shared medical records and a common database. We do not believe that characteristics of the healthcare system would bias the concentration or persistence over time of higher costs among individuals with more chronic conditions.

Although we used a standard classification of chronic conditions, another study limitation was that our identification source of chronic conditions was based on claims data rather than on medical records. This may have undercounted the prevalence of chronic conditions.

Implications for Practice, Policy, and Patients

The best care management approach for the workingage adult population may be a stratified approach. For most healthy (or presymptomatic) individuals, the primary focus is to keep them healthy. For individuals with 1 or 2 chronic conditions, the main focus is to maintain function and to prevent chronic conditions from progressing. Increasing the availability of nonvisit care and incorporating aspects of a medical home should foster this goal.

For the small group of individuals with multiple chronic conditions, more active and individualized management is necessary.

Focusing on chronic conditions commonly addressed by disease management programs, Charlson et al

found small cost differences among patients with only 1 chronic  condition, but costs increased rapidly as the number of comorbidities increased. They concluded that a patient-centered approach rather than a condition-centered approach is  needed to address the highest-cost patients with multiple comorbidities. With increasing prevalence of this type of patient, refocusing guidelines to meet patient-specific needs in managing multiple diseases may be required.

A traditional primary care model does not readily support a stratified approach to care delivery. Usual care delivery through in-office physician visits provides limited flexibility to address varying levels of need. A multidisciplinary team—based approach extending beyond the confines of the office could support the stratification of care to meet the varying needs of a population. For healthy patients, much care and management could be delivered in nonoffice settings by nonphysicians or via computer or phone.

Multiple chronic conditions among working-age adults lead to high costs over many years. Understanding how to effectively manage such patients is an important challenge.

James M. Naessens, ScD

Articles

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Simple Errors in Interpretation and Publication Can Be Costly

Risk-Stratification Methods for Identifying Patients for Care Coordination

Effect of Multiple Chronic Conditions Among Working-Age Adults