Paul Y. Takahashi, MD

With the advent of healthcare payment reform, identifying high-risk populations has become more important to providers. Existing risk-prediction models often focus on chronic conditions. This study sought to better understand other factors to improve identification of the highest risk population. 

A retrospective cohort study of a paneled primary care population utilizing 2010 data to calibrate a risk prediction model of hospital and emergency department (ED) use in 2011. 

Data were randomly split into development and validation data sets. We compared the enhanced model containing the additional risk predictors with the Minnesota medical tiering model. The study was conducted in the primary care practice of an integrated delivery system at an academic medical center in Rochester, Minnesota. The study focus was primary care medical home patients in 2010 and 2011 (n = 84,752), with the primary outcome of subsequent hospitalization or ED visit. A total of 42,384 individuals derived the enhanced risk-prediction model and 42,368 individuals validated the model. Predictors included Adjusted Clinical Groups—based Minnesota medical tiering, patient demographics, insurance status, and prior year healthcare utilization. Additional variables included specific mental and medical conditions, use of high-risk medications, and body mass index. 

The area under the curve in the enhanced model was 0.705 (95% CI, 0.698-0.712) compared with 0.662 (95% CI, 0.656-0.669) in the Minnesota medical tiering-only model. New high-risk patients in the enhanced model were more likely to have lack of health insurance, presence of Medicaid, diagnosed depression, and prior ED utilization. 

An enhanced model including additional healthcare-related factors improved the prediction of risk of hospitalization or ED visit.

		Many risk models that predict hospitalization or emergency department (ED) use are based on comorbid health burden. 

		We investigated other potential predictors, including previous utilization, and found an association of higher hospitalization and ED use with the following: Minnesota medical tiering, 2 prior ED visits, warfarin use, no insurance or Medicaid, being aged over 70 years, and obesity stage 3.

		These findings help to develop new models for determining patients at risk for adverse health outcomes. They focus on potential future areas of clinical practice and research on social determinants of health.

Changing the payment incentives to improve the organization of care delivery, including accountable care organizations, has led to a reorganization of delivery of care focusing on patient-centered medical homes.

 Many patient-centered medical homes stratify their populations and try to align the level of care with the needs of their patients. Providers manage the most complex patients with increased resources to avoid adverse outcomes requiring high-cost care in acute settings.

Two key questions need to be answered: Who is likely to have utilization, and what can be done for these patients? These medically complex patients could be better managed through care coordination programs that are integrated within the delivery system, such as Guided Care, which incorporates specially trained nurses to supplement primary care in a comprehensive care model that provides coordinated, patient-centered primary care for patients with complex needs.

 Different methods for identifying these patients have been developed, including provider identification and referral.

 A hybrid system that utilizes provider selection and an automated risk stratification model may be ideal. Most of the currently available public and proprietary risk models and instruments to identify the population for inclusion in more intensive management programs are based on number and combination of comorbidities. These instruments include self-administered tools such as the Predicting Risk of Admission (PRA) survey, which incorporates self-rated health and comorbid conditions as well as prior healthcare utilization.

 Another model utilizing medical record data is the Elder Risk Assessment (ERA) index, which determines risk of hospitalization based upon increasing age and previous hospitalization as well as comorbid health conditions.

A third risk model, Adjusted Clinical Groups (ACG), uses medical care claims data and stratifies patients based on age and existing chronic conditions.

 The state of Minnesota pays medical homes a care management fee based on Minnesota medical tiering. Minnesota medical tiering is a system of classifying and counting comorbidities originally based on collapsing chronic diseases from ACGs into a body system count.

 This system can also be manually applied without the proprietary software. Our clinical practice uses Minnesota medical tiering to identify appropriate patients for care management. Our prior work comparing these instruments, including Minnesota medical tiering, indicates that these instruments are similar in terms of predicting the risk of hospitalization or emergency department (ED) visit; however, the ACG performed slightly better while the Charlson Comorbidity Index performed worse.

 Although the administrative data-based risk identification methods have been validated and can predict hospitalization with C statistics of 0.67-0.73,

 other potential predictors may improve current risk prediction models and could provide information useful in developing tailored interventions to manage a patient’s condition.

Our clinical experience indicates that coordinating the management of multiple chronic conditions is important in reducing rehospitalizations among geriatric patients with multimorbidity, but other factors may also be important in predicting and managing an individual’s risk.

 In our practice, many patients predicted to be high risk are excluded from care management interventions because they have concurrent mental health or social conditions that care coordinators are unprepared to address. Body mass index (BMI), medication use,

 living environment, English as a second language, previous healthcare use including specialty visits, insurance, and mental health status

 might all be important factors. Patients with those issues may need a different approach, including more support from social services, pharmacy, and psychology.

Obesity is known to influence utilization, but it is widely underdiagnosed and so may be missed in many comorbidity counts.

 The use of anticoagulation therapy, pain, and diabetic medications can lead to an increased need for unplanned acute care and may be amenable to pharmacy consultation.

 Patients living on their own or in dysfunctional families may not be able to cope with increased treatment burdens and may benefit from social services.

 Similarly, those who need language interpretation services may need extra assistance. Patients with a high number of specialty visits may have uncoordinated care and may need help in coordinating their care. Lack of insurance and lower socioeconomic status are associated with increased rates of hospitalization.

 Patients experiencing mental health issues such as depression can also have higher readmission rates.

 Providers and systems with established comorbid health models should understand the potential roles of these additional predictors for hospitalization. 

It is unclear how well these additional factors would predict hospitalization when added to an already established clinical model based on administrative data. To answer this question, we performed a retrospective cohort study comparing a standard risk stratification model of Minnesota medical tiering with an enhanced model containing additional predictors of demographics, selected use of medications, mental health conditions, other specific chronic medical conditions, prior healthcare utilization, and insurance status. We sought to understand how well the enhanced model predicts hospitalization and ED use in a subsequent year.

We constructed a retrospective cohort study to develop and validate a clinical prediction model for hospitalization. The study was approved by the Mayo Clinic Institutional Review Board.

The study was conducted at an academic medical center in Rochester, Minnesota. The population came from the primary care medical practice, Employee and Community Health (ECH), which includes 4 primary care clinics in Dodge and Olmsted Counties, Minnesota. Primary care is provided by family medicine, primary care internal medicine, and community pediatric and adolescent medicine.

Patients 18 years or older who were assigned to a primary care provider within any of the primary care clinics for all 12 months in 2010 (base year) and throughout 12 months or until death in 2011 (assessment year) were included in the analysis. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law. Of the 90,411 patients within the ECH program, as of December 31, 2010, 84,752 (93.7%) gave consent for medical record review.

All information for enrolled ECH patients was electronically abstracted from the electronic medical record (EMR) and administrative databases for 2010 and 2011.

We modeled a combined binary outcome of hospitalization and/or ED visit (hereafter: hosp/ED) at St. Mary’s Hospital or Rochester Methodist Hospital in the assessment year. Hospitalization was defined as any elective or emergent overnight. To avoid double counting, ED visits exclude those resulting in a hospitalization.

Demographic variables—age, gender, marital status, and socioeconomic status&mdash;were collected at the end of the base year. Age was grouped into 6 categories: 18 to 29 years, 30 to 39 years, 40 to 49 years, 50 to 59 years, 60 to 69 years, and 70 years or older. Socioeconomic status was determined by insurance type (Medicaid or no insurance versus other insurance type).

As Minnesota medical tiering is the basis for care coordination payments for patient-centered medical homes in Minnesota, we used it as the basis for our risk model development. Medical conditions are based on diagnosis codes billed by clinical providers.

 Minnesota medical tiering has 5 categories from 0 with no conditions to tier 4 with more than 10 conditions. In our care management program, high-risk patients possess Minnesota medical-tiering scores of 3 or 4. 

Candidate predictors for identifying high-risk patients included primary speaking language not being English; high-risk medications (warfarin, insulin, narcotics); and previous healthcare utilization in specific settings (ED visits, hospitalization, specialty provider visits). Medications were based on the presence of outpatient pharmacy orders in 2010 written by any healthcare providers within the system. Medication use was determined from the EMR, not from filled prescriptions. The medications included warfarin, narcotics and insulin. Non—English speaking status was assigned to those patients needing an interpreter, which was documented within the electronic health record. We included all diagnosis codes (

International Classification of Diseases, Ninth Revision, Clinical Modification

]) from hospitalizations, ED visits, and primary and specialty care visits to identify comorbid mental health and medical conditions, based on a published method.

BMI—recorded as weight (kilograms) / height (meters)

—was also included as a potential predictor. BMI was assessed as being the most recent value in the record within 2 years prior to January 1, 2011. Specific BMI categories followed standard World Health Organization criteria: underweight (BMI <18.5); normal weight (BMI 18.5-24.9); overweight (BMI 25-29.9); stage 1 obesity (BMI 30-34.9); stage 2 obesity (BMI 35-39.9); and stage 3 obesity (BMI >40).

We developed the enhanced model to identify those patients at high risk of hosp/ED within the next year. The 2 models compared in the analysis consisted of Minnesota medical tiering and the enhanced model, which incorporated additional candidate factors. Data were randomly divided into development (50%) and validation cohorts.

We derived the final enhanced model from a stepwise logistic regression followed by a sensitivity analysis seeking stability in the predictors determined by the severity of the outcome variable. Predictors in the saturated model consisted of 72 main factors as well as all possible 1-way and 2-way interaction terms. We determined stability of the variables by changing the number of ED visits or inpatient (IP) visits in the outcome variable. We performed the saturated stepwise model while incrementally increasing the number of ED visits or IP visits. The final set of variables for each change in outcome variable was captured. We then considered a count of how many times a variable had a 

 value ≤.05 in the model. Finally, we included any variable that remained significant in 4 or more of the 11 models considered.

The stepwise regression was repeated 100 times and each repetition considered the variables in random order to determine if order affected the results of the stepwise algorithm. Each iteration produced the same model. Effects were entered into and removed from the model so that each forward selection step could be followed by a backward elimination step. The algorithm stopped when no further effects could be added to the model or if the current model was identical to a previous model.

Comparisons between prediction models focused on predictability and precision. We used C statistics (area under the receiver operating characteristic [ROC] curve [AUC]) as our measure of predictability. For assessment of precision, we compared the sets of high-risk patients found within the tier 3 and 4 groups in Minnesota medical tiering with an equal number of high-scoring patients from the enhanced model. Statistical analysis was performed using SAS version 9.3 (SAS Institute, Cary, North Carolina).

A total of 42,384 and 42,368 patients were included in the development and validation samples, respectively. The development and validation data sets were similar. There were no differences in sex, age decile, BMI, marital status, insurance status, prior healthcare utilization, comorbid health status, or Minnesota medical tiering (

). In addition, the 2 samples had similar rates of hospitalization and ED visits with 18.2% of the development sample and 17.6% of the validation sample having a hospitalization or ED visit in the subsequent year.

Those patients who belonged to the highest category of Minnesota medical tiering (4) had the greatest odds of incurring any hospitalization or ED visit with an odds ratio of 18.2 (95% CI, 15.6-21.24) compared with the lowest tier category (0). Next most important was prior utilization of 2 or more ED visits with an odds ratio of 7.13 (95% CI, 6.34-8.01) for predicting hosp/ED. Similarly, 2 or more previous hospital stays had an odds ratio of 4.28 (95% CI, 3.90-4.69) for predicting hosp/ED. Mental health conditions, including substance abuse, personality disorder or dementia, each had odds near 3 for predicting hosp/ED. Specifically, depression had an odds ratio of 2.92 (95% CI, 2.71-3.15). Among other medical conditions, cardiac disease with heart failure and pulmonary heart disease had odds ratios over 5. All specific conditions with odds ratios over 2 are reported in Table 1. All specific conditions are listed in 

. Patients in each of the medication categories had odds ratios greater than 2, as did those utilizing 3 or more specialist visits in the prior year. Patients with a language interpreter did not have elevated risk of hosp/ED (Table 2).

The multivariable enhanced tiering model is presented in 

. The final model included 30 main effects (representing 16 factors) and 4 2-way interaction terms (more detailed results are available from the authors).

Of notable mention, being in the highest medical tier remains the strongest predictor for hosp/ED after the addition of other factors with an odds ratio of 3.50 (95% CI, 2.78-4.42). More than 1 previous ED visit also resulted in an increased odds ratio of 3.17 (95% CI, 2.78-3.60). Several significant interaction terms are noted in Table 3. In the validation cohort, the C statistic for the enhanced multivariable model was 0.711 (95% CI, 0.704-0.717) compared with a C statistic for the Minnesota medical tiering alone model of 0.667 (95% CI 0.660-0.673). These C statistics were actually slightly higher in the validation cohort than in the development cohort (enhanced model: 0.705 [95% CI, 0.698-0.712]; Minnesota medical tiering: 0.662 [95% CI, 0.656-0.669]). The differences between the ability of the enhanced model to predict hospitalization and ED use and Minnesota medical tiering are displayed in the ROC curves found in the 

Many different patients were identified as being high-risk in the enhanced model as compared with those in the original Minnesota medical tiering model (tier 3 or 4). 

 presents the characteristics of high-risk patients identified with both models as well as those unique to the 2 approaches. We found that patients identified as high-risk in the enhanced model were much more likely to experience hospital utilization in 2011 than those in the Minnesota medical tiering-only model (

 <.001). Patients identified as high-risk based only on the enhanced model, compared with those identified only through Minnesota medical tiering, were of younger age, had higher previous healthcare utilization, and had more frequent mental health conditions. Many of the high-risk enhanced model patients were aged less than 40 years compared with only 11% of patients being aged less than 40 years in the Minnesota medical tiering model.

In this retrospective cohort of patients receiving care within the ECH, several variables were important in predicting hosp/ED in addition to the number of chronic comorbidities currently used in Minnesota medical tiering and our care coordination identification. We found that age, BMI, previous healthcare utilization, mental health conditions, anemia, heart failure, epilepsy, hyperlipidemia, warfarin use, and narcotic use all simultaneously contributed to the prediction of hospitalization and ED visits in the next year. This enhanced model had an AUC of 0.711 (95% CI, 0.704-0.717) for predicting hosp/ED, as compared with 0.667 (95% CI, 0.660-0.673) found in the model utilizing only Minnesota medical tiering. Previous work comparing different risk stratification methods for hospitalization determined slight superiority of ACG compared with the ERA, Hierarchical Condition Categories, Minnesota medical tiering, and chronic care counting instruments.

 The enhanced model takes into consideration previous hospitalization, specific high-risk illnesses, mental health conditions, and high-risk medication use (eg, warfarin, narcotics) that are not universally accounted for in other models. In comparing high-risk groups of the same size from both models and excluding patients identified in both models, 47% of the patients in the enhanced model experienced hosp/ED compared with 32.5% of patients from the Minnesota medical tiering model. The enhanced model accounts for factors in a younger population (eg, mental health and previous utilization) that are not detected in the Minnesota medical tiering model.

Previous hospital and ED utilization served as strong predictors for hosp/ED in the following year. Specifically, 2 or more prior ED visits presented an odds ratio (OR) close to 3 while 1 prior hospital stay resulted in an OR of 1.79 (95% CI, 1.66-1.93). Together, hospitalization and ED visits often indicate a medically complex individual or an individual with severe illness. Many risk stratification models have utilized previous healthcare usage as a predictor for future healthcare use. The PRA incorporates previous hospitalization within the past year as a factor predictive of future hospitalization.

 Similarly, the ERA also includes previous hospital days to calculate a risk score for future hospitalization.

 Previous ED visits as well as previous hospitalization have been associated with increased hospital admission in a multi-centered VA study of 1378 patients designed to identify factors of hospitalization.

Articles

An enhanced risk model incorporating medication use, prior healthcare utilization, and mental health with comorbid health conditions predicts healthcare utilization better than health conditions alone.  

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Care coordination is a key component of the patient-centered medical home. However, the mechanism for identifying primary care patients who may benefit the most from this model of care is unclear.

To evaluate the performance of several risk-adjustment/stratification instruments in predicting healthcare utilization.

 All adults empaneled in 2009 and 2010 (n = 83,187) in a primary care practice were studied. We evaluated 6 models: Adjusted Clinical Groups (ACGs), Hierarchical Condition Categories (HCCs), Elder Risk Assessment, Chronic Comorbidity Count, Charlson Comorbidity Index, and Minnesota Health Care Home Tiering. A seventh model combining Minnesota Tiering with ERA score was also assessed. Logistic regression models using demographic characteristics and diagnoses from 2009 were used to predict healthcare utilization and costs for 2010 with binary outcomes (emergency department [ED] visits, hospitalizations, 30-day readmissions, and highcost users in the top 10%), using the C statistic and goodness of fit among the top decile.

The ACG model outperformed the others in predicting hospitalizations with a C statistic range of 0.67 (CMS-HCC) to 0.73. In predicting ED visits, the C statistic ranged from 0.58 (CMSHCC) to 0.67 (ACG). When predicting the top 10% highest cost users, the performance of the ACG model was good (area under the curve = 0.76) and superior to the others.

 Although ACG models generally performed better in predicting utilization, use of any of these models will help practices implement care coordination more efficiently.

For care coordination to be used effectively, it is important to identify which patients are likely to benefit from it. Little evidence exists as to which risk-stratification method performs better in predicting hospitalizations, emergency department visits, 30-day readmissions, and high-cost users.

		There appeared to be good concordance among 6 different risk screening instruments in predicting hospitalization.

		Adjusted Clinical Group classification was generally better than other models in predicting healthcare utilization.n

		Focusing care coordination efforts within the medical home on patients likely to benefit most requires appropriate identification of the highest risk, highest utilizing patients.

With rising prevalence of chronic disease, patient-centered medical homes (PCMHs) were created to deliver higherquality, more cost-effective primary care in the United States.

 A key component of PCMH is the implementation of care coordination, especially if targeted to the right people.

 While some states have developed risk-adjusted care coordination payments,  methods for effective identification of individuals likely to benefit from care coordination are not clear.

Patients with multiple chronic conditions may see many providers, often in an uncoordinated manner, adding to their increased risk of high healthcare utilization through test duplication, edication management conflicts, and medical errors.

These patients are also at increased risk for hospital admissions and emergency department (ED) visits. Primary care doctors may not routinely have the time or resources to properly manage these complex patients who have multiple chronic health problems. Thus, care  coordination for the right patients could decrease unnecessary care and prevent adverse outcomes.

Most existing PCMH programs identify care coordination patients through physician referrals.

 However, this method may not identify all potential beneficiaries. Care coordination should be focused on patients who will benefit most, maximizing the impact on both quality and costs. Risk-stratification models can be efficient tools for both providers and health plans to screen populations and select individuals for care management programs who are at risk of future hospitalization and functional decline.

 Several methods using administrative data have been described to help with the identification of high-risk  patients.

 However, it remains unclear which method will perform best in identifying patients in need of care coordination. To answer this important and fundamental question, we compared the performance of 6 common risk-adjustment methods in predicting hospitalizations, ED visits, 30-day readmissions, and high expenditures. This study extends the existing literature on patient selection for care coordination with results from direct comparison of the following models: Adjusted Clinical Groups (ACGs), Hierarchical Condition Categories (HCCs), Elder Risk Assessment (ERA), Chronic Comorbidity Count (CCC), Charlson Comorbidity Index, and Minnesota Health Care Home Tiering (MN Tiering). The identification of an optimal method may guide implementation of care coordination programs within PCMHs. Furthermore, identifying these predictable  high-utilization cases may lead to other effective interventions.

A retrospective cohort analysis compared multiple methods to identify high-risk patients. These methods were applied to all primary care patients 18 years or older empaneled to the Employee and Community Health (ECH) practice (family medicine, primary care internal medicine, and community pediatric and adolescent medicine) at Mayo Clinic, Rochester, Minnesota, the community-focused primary care arm of a large integrated multispecialty group practice.

To be a patient of ECH, one must be a Mayo Clinic employee or live in the community, identify an ECH primary care provider, and be enrolled within his/her panel. Participants assigned to a primary care provider for all 12 months in 2009 (base year) and throughout 12 months or until death in 2010 (assessment year) based on the electronic medical record were included in the analysis. The organized development of care coordination within ECH occurred during 2011; hence, usual care (encounter driven) was provided to all patients during the study. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law.

All information was electronically abstracted from the electronic medical record and administrative databases within Mayo Clinic’s health records system. Mayo Clinic maintains all electronic medical record information within 1 system, including inpatient and outpatient visit information.

Demographic variables collected in the base year included age, sex, marital status, and insurance status. Age was grouped into 4 categories of 18 to 44 years, 45 to 64 years, 65 to 84 years, and more than 85 years. Insurance status was also grouped into 4 categories: public (Medicare or Medicaid), private, no insurance, and unknown. Diagnosis codes (

International Classification of Diseases, Ninth Revision, Clinical Modification 

) for each patient encounter, as well as utilization and cost information, were extracted from institutional billing data for year 2009. Number of hospital days was also retrieved from 2008 to calculate ERA scores. We included all diagnosis codes from hospitalizations, ED visits, and primary and specialty care evaluation and management visits.

The ACG is a commonly used classification system based on administrative diagnosis data and developed by Johns Hopkins to measure morbidity.

 The ACG methodology was developed to predict the utilization of medical resources using the presence or absence of specific diagnoses from both inpatient and outpatient services for a specified period of time, along with age and sex, to classify each person into 1 of 93 discrete ACG categories with similar expected utilization patterns. Adjusted Clinical Groups can be used to improve accuracy and fairness in forecasting healthcare utilization

 and have been found to predict inpatient hospitalizations as well as or better than other case-mix tools in many health systems.

 The MN Tiering model is based on a product of ACGs, Major Extended Diagnostic Groups.

 MN Tiering is currently being used by the State of Minnesota to determine management fees for care coordination among medical home plans. The purpose of MN Tiering is to group patients into “complexity tiers” based on the number of major condition categories from which they suffer.3 The total sum of conditions is grouped into the following 5 patient complexity levels: low (tier 0): 0  conditions; basic (tier 1): 1 to 3 conditions; intermediate (tier 2): 4 to 6 conditions; extended (tier 3): 7 to 9 conditions; and complex (tier 4): 10 or more conditions.

In 2004, the Centers for Medicare & Medicaid Services (CMS) implemented HCCs to adjust Medicare capitation payments for health expenditure risk of Medicare Advantage Plan (health maintenance organization) enrollees.

 diagnosis codes and demographic data for each patient are aggregated into 70 condition categories that contribute to a single risk score.17 Several studies among Medicare patients have provided evidence that HCC scores for risk adjustment can be effective at predicting hospitalizations.

 Elder Risk Assessment Index. The ERA Index was developed to identify patients at risk for hospitalization and ED visits in adults 60 years or older.

The ERA Index incorporates a weighted score of age, sex, number of hospital days in the prior 2 years, and marital status, as well as selected medical conditions (diabetes, coronary artery disease, congestive heart failure, stroke, chronic obstructive pulmonary disease,and dementia).

The minimum score on the index is —1 and the maximum score possible is 34.

The CCC method was derived by Naessens and colleagues

 from a modification of the method of Hwang and colleagues,

 based on the publicly available Agency for Healthcare Research and Quality’s Clinical Classification Software. The CCC method is more comprehensive than most comorbidity counts. The total sum of chronic conditions was grouped into 6 categories: 0, 1, 2, 3, 4, and 5 or more. Comorbidity counts have been shown to be associated with high annual costs as well as persistence in high costs.

 The Charlson Comorbidity Index, originally derived to classify comorbidities affecting 1-year mortality in cancer patients, sums weights for 17 specific conditions.23 In our study, the sum of Charlson Comorbidity Index counts was used to predict future outcomes. The performance of the Charlson Comorbidity Index in predicting poor outcomes has been assessed in various large populations, and its validity as a prognostic measure of outcomes has been consistently demonstrated.

 A care coordination patient enrollment process was developed in our primary care practice based on combining MN Tiering and the ERA score as a hybrid model. Patients eligible for enrollment in care coordination included all patients in MN tier 4 (individuals with more than 10 comorbid conditions) and those in MN tier 3 with ERA scores greater than 10. Although all of the previously mentioned risk-stratification instruments have been validated, it remains unclear how the hybrid method of combining MN Tiering and ERA score will compare with them.

Several outcome measures could be used as surrogates for patients who may benefit from care coordination. We modeled 4 binary outcomes reflecting high utilization within the 12-month prediction period of 2010: (1) any inpatient hospitalization, (2) ED visits not resulting in hospitalizations, (3) any readmission within 30 days of an initial hospitalization, and (4) being a high-cost user. A high-cost user wasdefined using the threshold of the top 10% users in 2010 (>$7457). We chose to focus on the top 10% of users because they consume almost 70% of total costs, but are less influenced by catastrophic conditions and potentially more amenable to the effects of care coordination than patients at a higher threshold. These outcomes are utilization measures believed to be influenced by effective care coordination. 

ED visits resulting in a hospitalization were recorded as a single inpatient hospitalization event. Total costs reflect a provider perspective and were based on standardized inflation-adjusted cost estimates for each service or procedure provided within our healthcare system (not including outpatient pharmacy services) in 2010 constant Medicare dollars.

We applied logistic regression models to compare the 6 risk-stratification methods to determine which method best predicted the dichotomous outcomes in the subsequent year. Model performance was based on explanatory power and goodness of fit. Explanatory power was assessed by using the C statistic with 95% confidence intervals to predict (1) hospitalizations, ED visits, 30-day readmissions, and high-cost users and (2) the ability of each model to identify individuals with the outcomes of interest in the highest and lowest predicted deciles. The C statistic is a measure of model discrimination and is equivalent to the area under the receiver operating characteristic curve.

To assess goodness of fit, we compared the observed and predicted hospitalizations, ED visits, readmissions, and high-cost users in the lowest and highest deciles on the basis of predicted probabilities.

To address calibration, we further focused our assessment on those patients at the highest end of each risk score to identify which patients need care coordination. Since the hybrid model was used clinically in 2011, we used the number of patients considered for PCMH enrollment as a target number to establish the threshold for potential care coordination (based on the highest estimated probability of hospitalizations). We then determined (1) how much overlap in identification occurred in each of the 7 methods and (2) how much of the total utilization the 6 other approaches would have identified. Analyses were conducted with SAS software, version 9.1 (SAS Institute Inc, Cary, North Carolina). The study was approved by the Mayo Clinic Institutional Review Board. No external funding was obtained.

The study population included 83,187 patients who met inclusion criteria between January 1, 2009, and December 31, 2010. The mean age of the base population was 46.9 years, 54.6% were female, 63.1% had private insurance, and 21.8%

). Table 1 shows the frequency distributions of the demographic characteristics at the end of 2009, as well as the percentage of paneled patients with selected chronic diseases.

Healthcare utilization and resource use for the base and prediction years were similar (Table 1). Approximately 8% of patients had a hospitalization, and 13% percent of the cohort had an ED visit. Compared with the base year 2009, the mean total cost in 2010 was nearly the same, decreasing by 1%. We saw the expected concentration of healthcare services among a relatively small number of individuals (

). Overall, 32.4% of the most expensive 10% of patients in 2009 were also in the top 10% of patients in 2010. Furthermore, our outcomes of interest were correlated, though not perfectly. High cost was associated with each of the utilization measures, with  Pearson correlations ranging from 0.22 for ED visits to 0.72 for hospitalizations. The correlations between utilization measures ranged from 0.12 between ED visits and readmission to 0.35 between hospitalization and readmission.

, the ACG model outperformed the other 5 models in predicting hospitalizations, with a C statistic range of 0.67 (CMS-HCC) to 0.73 (ACG). The ERA score and MN Tiering followed close behind ACG for prediction of hospitalization (C statistic = 0.71). In the models predicting ED visits, the C statistic ranged from 0.58 (CMS-HCC) to 0.67, with the ACG model having the best predictive ability. Further, the ACG model outperformed other models when predicting 30-day readmissions; the C statistic ranged from 0.74 (CMS-HCC) to 0.81 (ACG). When predicting healthcare expenditures for the top 10% high-cost users, the performance of the ACG model was good (C statistic = 0.76) and superior to that of other models. CMS-HCC had the lowest predictive ability for all 4 outcomes. It is important to point out that although ACGs had the best predictive ability, much of the variability in outcomes was unexplained by any model. For each outcome, models with higher C statistics also had higher rates of actual events among the highest deciles. For example, the top decile for ACG models had 27% with a hospitalization and 31% with at least 1 ED visit, whereas the top decile for CMS-HCC models had 25% with a hospitalization and 23% with an ED visit. There was more potential discrimination with ACG.

To further evaluate the 6 methods, we compared the patients in the top decile of predicted probability of having a hospitalization (

). The ERA method tended to overpredict for those in the top decile, whereas the CMSHCC method underpredicted. For the ACG, MN Tiering, and CCC methods, the actual and predicted hospitalizations were nearly equivalent. Using the model based on ACGs, 26.8% of patients in the highest decile were hospitalized; using the MN Tiering model, 25.1% in the highest decile were hospitalized; and using the Charlson Comorbidity Index, only 22.9% in the highest decile were hospitalized. Similar results were seen for the other 3 outcomes (results are available in 

A total of 2347 (2.8%) patients were identified as meriting care coordination based on the hybrid clinical approach (Table 3). At least 40% of these base patients were in the selected top group of patients, irrespective of which risk method was used. Interestingly, our initial clinical implementation actually identified the patients with the highest number of hospitalizations, the highest percentage of patients with any ED visit, and the patients with highest total costs.

We assessed 6 risk instrument methods based on administrative and demographic data. We evaluated the performance of the 6 models against one another to assess the ability to predict future healthcare utilization. We concluded that the ACGs produced a more accurate prediction of future healthcare utilization relative to the other models.

All risk prediction models for hospitalization had fair predictive value, with ACG having the highest overall predictive C statistic at 0.73 and the HCC model having the lowest predictive C statistic at 0.67. In a previous large study, the ACG had an excellent predictive area under the curve of 0.80.29 An earlier study on the HCC showed an area under the curve of 0.638 for predicting hospitalizations among newly enrolled Medicare patients.

MN Tiering, CCC, and the ERA all performed similarly; thus, use of each could be justified. These novel and unique findings indicate to both providers and health plans that any of these risk-stratification models can be used for clinical purposes. The individual risk instruments performed in a similar fashion for ED visits, readmissions, and high-cost users as well as hospitalizations, as might be expected, because these outcomes tend to be correlated. The predictive values of the risk-stratification instruments were slightly lower for ED use, but higher for 30-day readmission and high-cost users. These findings provide important information regarding use of newer risk-stratification tools like MN Tiering, ERA, and CCC. The instruments predicted not only hospitalization but also re-hospitalization and ED visits.

Although all rely on the presence of diagnoses and demographic factors, the 6 risk-screening instruments vary with respect to ease of implementation. Unfortunately, the best performing methods, ACG and MN Tiering, are also the only methods we examined that require software licensing. A form of MN Tiering based on manual classification is available.

CMS-HCC is a software package that can be downloaded from CMS. The algorithms for the ERA Index, CCC, and Charlson Comorbidity Index have all been published and are available, but need to be programmed to be applied for local clinical use.

Identifying which patients are likely to benefit from care coordination is important. We evaluated the performance of 6 risk-screening instruments in predicting healthcare utilization.

Paul Y. Takahashi, MD

Articles

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Risk-Stratification Methods for Identifying Patients for Care Coordination