Currently Viewing:
The American Journal of Managed Care July 2016
Currently Reading
Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home
Paul Y. Takahashi, MD; Herbert C. Heien, MS; Lindsey R. Sangaralingham, MPH; Nilay D. Shah, PhD; and James M. Naessens, ScD
Choosing Wisely Campaign Builds Momentum
Daniel B. Wolfson, MHSA
Insurance Coverage and Diabetes Quality Indicators Among Patients in NHANES
Emily D. Doucette, MD; Joanne Salas, MPH; and Jeffrey F. Scherrer, PhD
Shared Medical Appointments: Balancing Efficiency With Patient Satisfaction and Outcomes
Shelly P. Smith, DNP, APRN-BC, and Beth L. Elias, PhD, MS
Changes in Premiums of Cancelled Nongroup Plans Under the Affordable Care Act
Jared Lane K. Maeda, PhD, MPH; Jersey Chen, MD, MPH; and Brent R. Plemons, BS
Economic and Clinical Impact of Routine Weekend Catheterization Services
Kirsten Hall Long, PhD; James P. Moriarty, MSc; Jeanine E. Ransom, BA; Ryan J. Lennon, MS; Verghese Mathew, MD; Rajiv Gulati, MD, PhD; Gurpreet S. Sandhu, MD, PhD; and Charanjit S. Rihal, MD, MBA
Hospital Participation in ACOs Associated With Other Value-Based Program Improvement
David Muhlestein, PhD, JD; Tianna Tu, BA; Katelyn de Lisle, BS; and Thomas Merrill, BA
A Restricted Look at CRC Screening: Not Considering Annual Stool Testing as an Option
Karen M. Kuntz, ScD; Ann G. Zauber, PhD; Amy B. Knudsen, PhD; Carolyn M. Rutter, PhD; Iris Lansdorp-Vogelaar, PhD; Barry M. Berger, MD FCAP; and Bernard Levin, MD, FACP
Impact of a Physician-Led Point of Care Medication Delivery System on Medication Adherence
Ana Palacio, MD, MPH; Jessica Chen, MD; Leonardo Tamariz, MD, MPH; Sylvia D. Garay, MD; Hua Li, PhD; and Olveen Carrasquillo, MD, MPH
Testing of a Tethered Personal Health Record Framework for Early End-of-Life Discussions
Seuli Bose-Brill, MD; Matthew Kretovics, MPH; Taylor Ballenger, MD; Gabriella Modan, PhD; Albert Lai, PhD; Lindsay Belanger, MPH; Stephen Koesters, MD; Taylor Pressler-Vydra, MS; Christopher Holloman, PhD; and Celia Wills, PhD, RN

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Paul Y. Takahashi, MD; Herbert C. Heien, MS; Lindsey R. Sangaralingham, MPH; Nilay D. Shah, PhD; and James M. Naessens, ScD
An enhanced risk model incorporating medication use, prior healthcare utilization, and mental health with comorbid health conditions predicts healthcare utilization better than health conditions alone.


Objectives: With the advent of healthcare payment reform, identifying high-risk populations has become more important to providers. Existing risk-prediction models often focus on chronic conditions. This study sought to better understand other factors to improve identification of the highest risk population.

Study Design: A retrospective cohort study of a paneled primary care population utilizing 2010 data to calibrate a risk prediction model of hospital and emergency department (ED) use in 2011.

Methods: Data were randomly split into development and validation data sets. We compared the enhanced model containing the additional risk predictors with the Minnesota medical tiering model. The study was conducted in the primary care practice of an integrated delivery system at an academic medical center in Rochester, Minnesota. The study focus was primary care medical home patients in 2010 and 2011 (n = 84,752), with the primary outcome of subsequent hospitalization or ED visit. A total of 42,384 individuals derived the enhanced risk-prediction model and 42,368 individuals validated the model. Predictors included Adjusted Clinical Groups–based Minnesota medical tiering, patient demographics, insurance status, and prior year healthcare utilization. Additional variables included specific mental and medical conditions, use of high-risk medications, and body mass index.

Results: The area under the curve in the enhanced model was 0.705 (95% CI, 0.698-0.712) compared with 0.662 (95% CI, 0.656-0.669) in the Minnesota medical tiering-only model. New high-risk patients in the enhanced model were more likely to have lack of health insurance, presence of Medicaid, diagnosed depression, and prior ED utilization.

Conclusions: An enhanced model including additional healthcare-related factors improved the prediction of risk of hospitalization or ED visit.

Am J Manag Care. 2016;22(7):475-483

Take-Away Points
  • Many risk models that predict hospitalization or emergency department (ED) use are based on comorbid health burden. 
  • We investigated other potential predictors, including previous utilization, and found an association of higher hospitalization and ED use with the following: Minnesota medical tiering, 2 prior ED visits, warfarin use, no insurance or Medicaid, being aged over 70 years, and obesity stage 3.
  • These findings help to develop new models for determining patients at risk for adverse health outcomes. They focus on potential future areas of clinical practice and research on social determinants of health.
Changing the payment incentives to improve the organization of care delivery, including accountable care organizations, has led to a reorganization of delivery of care focusing on patient-centered medical homes.1 Many patient-centered medical homes stratify their populations and try to align the level of care with the needs of their patients. Providers manage the most complex patients with increased resources to avoid adverse outcomes requiring high-cost care in acute settings.

Two key questions need to be answered: Who is likely to have utilization, and what can be done for these patients? These medically complex patients could be better managed through care coordination programs that are integrated within the delivery system, such as Guided Care, which incorporates specially trained nurses to supplement primary care in a comprehensive care model that provides coordinated, patient-centered primary care for patients with complex needs.2 Different methods for identifying these patients have been developed, including provider identification and referral.3 A hybrid system that utilizes provider selection and an automated risk stratification model may be ideal. Most of the currently available public and proprietary risk models and instruments to identify the population for inclusion in more intensive management programs are based on number and combination of comorbidities. These instruments include self-administered tools such as the Predicting Risk of Admission (PRA) survey, which incorporates self-rated health and comorbid conditions as well as prior healthcare utilization.4 Another model utilizing medical record data is the Elder Risk Assessment (ERA) index, which determines risk of hospitalization based upon increasing age and previous hospitalization as well as comorbid health conditions.5

A third risk model, Adjusted Clinical Groups (ACG), uses medical care claims data and stratifies patients based on age and existing chronic conditions.6 The state of Minnesota pays medical homes a care management fee based on Minnesota medical tiering. Minnesota medical tiering is a system of classifying and counting comorbidities originally based on collapsing chronic diseases from ACGs into a body system count.7 This system can also be manually applied without the proprietary software. Our clinical practice uses Minnesota medical tiering to identify appropriate patients for care management. Our prior work comparing these instruments, including Minnesota medical tiering, indicates that these instruments are similar in terms of predicting the risk of hospitalization or emergency department (ED) visit; however, the ACG performed slightly better while the Charlson Comorbidity Index performed worse.8 Although the administrative data-based risk identification methods have been validated and can predict hospitalization with C statistics of 0.67-0.73,8 other potential predictors may improve current risk prediction models and could provide information useful in developing tailored interventions to manage a patient’s condition.

Our clinical experience indicates that coordinating the management of multiple chronic conditions is important in reducing rehospitalizations among geriatric patients with multimorbidity, but other factors may also be important in predicting and managing an individual’s risk.9 In our practice, many patients predicted to be high risk are excluded from care management interventions because they have concurrent mental health or social conditions that care coordinators are unprepared to address. Body mass index (BMI), medication use,10 living environment, English as a second language, previous healthcare use including specialty visits, insurance, and mental health status11 might all be important factors. Patients with those issues may need a different approach, including more support from social services, pharmacy, and psychology.

Obesity is known to influence utilization, but it is widely underdiagnosed and so may be missed in many comorbidity counts.12 The use of anticoagulation therapy, pain, and diabetic medications can lead to an increased need for unplanned acute care and may be amenable to pharmacy consultation.13 Patients living on their own or in dysfunctional families may not be able to cope with increased treatment burdens and may benefit from social services.14,15 Similarly, those who need language interpretation services may need extra assistance. Patients with a high number of specialty visits may have uncoordinated care and may need help in coordinating their care. Lack of insurance and lower socioeconomic status are associated with increased rates of hospitalization.16 Patients experiencing mental health issues such as depression can also have higher readmission rates.7,17 Providers and systems with established comorbid health models should understand the potential roles of these additional predictors for hospitalization. 

It is unclear how well these additional factors would predict hospitalization when added to an already established clinical model based on administrative data. To answer this question, we performed a retrospective cohort study comparing a standard risk stratification model of Minnesota medical tiering with an enhanced model containing additional predictors of demographics, selected use of medications, mental health conditions, other specific chronic medical conditions, prior healthcare utilization, and insurance status. We sought to understand how well the enhanced model predicts hospitalization and ED use in a subsequent year.


We constructed a retrospective cohort study to develop and validate a clinical prediction model for hospitalization. The study was approved by the Mayo Clinic Institutional Review Board.


The study was conducted at an academic medical center in Rochester, Minnesota. The population came from the primary care medical practice, Employee and Community Health (ECH), which includes 4 primary care clinics in Dodge and Olmsted Counties, Minnesota. Primary care is provided by family medicine, primary care internal medicine, and community pediatric and adolescent medicine.

Patient Population

Patients 18 years or older who were assigned to a primary care provider within any of the primary care clinics for all 12 months in 2010 (base year) and throughout 12 months or until death in 2011 (assessment year) were included in the analysis. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law. Of the 90,411 patients within the ECH program, as of December 31, 2010, 84,752 (93.7%) gave consent for medical record review.

Data Collection

All information for enrolled ECH patients was electronically abstracted from the electronic medical record (EMR) and administrative databases for 2010 and 2011.

Outcomes and Predictors

We modeled a combined binary outcome of hospitalization and/or ED visit (hereafter: hosp/ED) at St. Mary’s Hospital or Rochester Methodist Hospital in the assessment year. Hospitalization was defined as any elective or emergent overnight. To avoid double counting, ED visits exclude those resulting in a hospitalization.

Demographic variables—age, gender, marital status, and socioeconomic status—were collected at the end of the base year. Age was grouped into 6 categories: 18 to 29 years, 30 to 39 years, 40 to 49 years, 50 to 59 years, 60 to 69 years, and 70 years or older. Socioeconomic status was determined by insurance type (Medicaid or no insurance versus other insurance type).

As Minnesota medical tiering is the basis for care coordination payments for patient-centered medical homes in Minnesota, we used it as the basis for our risk model development. Medical conditions are based on diagnosis codes billed by clinical providers.18 Minnesota medical tiering has 5 categories from 0 with no conditions to tier 4 with more than 10 conditions. In our care management program, high-risk patients possess Minnesota medical-tiering scores of 3 or 4. 

Candidate predictors for identifying high-risk patients included primary speaking language not being English; high-risk medications (warfarin, insulin, narcotics); and previous healthcare utilization in specific settings (ED visits, hospitalization, specialty provider visits). Medications were based on the presence of outpatient pharmacy orders in 2010 written by any healthcare providers within the system. Medication use was determined from the EMR, not from filled prescriptions. The medications included warfarin, narcotics and insulin. Non–English speaking status was assigned to those patients needing an interpreter, which was documented within the electronic health record. We included all diagnosis codes (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM]) from hospitalizations, ED visits, and primary and specialty care visits to identify comorbid mental health and medical conditions, based on a published method.19 Each outpatient visit had 4 potential ICD-9-CM codes and inpatient visits had 25 ICD-9-CM codes.

BMI—recorded as weight (kilograms) / height (meters)2—was also included as a potential predictor. BMI was assessed as being the most recent value in the record within 2 years prior to January 1, 2011. Specific BMI categories followed standard World Health Organization criteria: underweight (BMI <18.5); normal weight (BMI 18.5-24.9); overweight (BMI 25-29.9); stage 1 obesity (BMI 30-34.9); stage 2 obesity (BMI 35-39.9); and stage 3 obesity (BMI >40).20

Statistical Analysis and Model Building

We developed the enhanced model to identify those patients at high risk of hosp/ED within the next year. The 2 models compared in the analysis consisted of Minnesota medical tiering and the enhanced model, which incorporated additional candidate factors. Data were randomly divided into development (50%) and validation cohorts.

We derived the final enhanced model from a stepwise logistic regression followed by a sensitivity analysis seeking stability in the predictors determined by the severity of the outcome variable. Predictors in the saturated model consisted of 72 main factors as well as all possible 1-way and 2-way interaction terms. We determined stability of the variables by changing the number of ED visits or inpatient (IP) visits in the outcome variable. We performed the saturated stepwise model while incrementally increasing the number of ED visits or IP visits. The final set of variables for each change in outcome variable was captured. We then considered a count of how many times a variable had a P value ≤.05 in the model. Finally, we included any variable that remained significant in 4 or more of the 11 models considered.

The stepwise regression was repeated 100 times and each repetition considered the variables in random order to determine if order affected the results of the stepwise algorithm. Each iteration produced the same model. Effects were entered into and removed from the model so that each forward selection step could be followed by a backward elimination step. The algorithm stopped when no further effects could be added to the model or if the current model was identical to a previous model.

Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up