Risk Stratification for Return Emergency Department Visits Among High-Risk Patients

Katherine E.M. Miller, MSPH; Wei Duan-Porter, MD, PhD; Karen M. Stechuchak, MS; Elizabeth Mahanna, MPH; Cynthia J. Coffman, PhD; Morris Weinberger, PhD; Courtney Harold Van Houtven, PhD; Eugene Z. Oddone, MD, MHSc; Katina Morris, MS; Kenneth E. Schmader, MD; Cristina C. Hendrix, DNS, GNP-BC; Chad Kessler, MD; and Susan Nicole Hastings, MD, MHSc

Emergency department (ED) utilization not resulting in hospital admission, referred to as outpatient ED visits, may be avoidable1 and more costly than an outpatient clinic2; thus, it is considered potentially low-value care. To reduce low-value care, risk prediction models have been developed to identify the patients who account for a disproportionately large amount of healthcare utilization; the goal is to target these patients for interventions that can reduce avoidable utilization.3 Researchers often develop and validate risk prediction models for disease-specific populations3; however, the models may not be generalizable to broader and more medically complex populations. 

Outpatient ED visits are common in the Veterans Health Administration (VHA), the largest integrated healthcare system in the United States, which serves more than 9 million veterans nationally.4 From 2007 to 2008, 80% of ED visits were outpatient; of these, 15% had a repeat ED visit within 30 days—a higher rate than Medicare beneficiaries.1 The VHA has been at the forefront of predictive analytics in healthcare and has implemented Care Assessment Need (CAN) scores for all VHA users. CAN scores use complex multivariate modeling to generate a validated risk prediction of hospitalization and/or death within 90 days or 1 year, using available electronic health records (EHRs) and administrative data.5 CAN scores are utilized to optimize care coordination and resource allocation for high-risk veterans.5 

However, it is unknown whether CAN scores identify patients at higher risk for repeat ED utilization—especially compared with simpler strategies, such as a previous history of high ED utilization.5-11 Thus, in this exploratory study, we examined whether the CAN score provided further information on risk for repeat ED visits for a high-risk cohort of VHA-affiliated patients. First, we compared whether CAN scores and Super User status (ie, having 4 or more ED visits within the last year)12 identified the same patients as high risk. Then, we assessed whether these risk classifications could predict repeat ED visits that occurred within 90 days of an index ED visit. 


Study Cohort

The study cohort met initial eligibility criteria for an ongoing randomized clinical trial at the Durham Veterans Affairs Medical Center (DVAMC), Discharge Information and Support for Patients Receiving Outpatient Care in the ED (DISPO ED), which took place from March 10 to September 30, 2014.13 DISPO ED examined the effectiveness of a nurse-led intervention to reduce repeat ED visits. In addition to having an index outpatient ED visit, inclusion criteria included: 1 or more visits to a DVAMC-affiliated primary care clinic within the previous 12 months (proxy for engagement with the VA system), 1 or more DVAMC ED visit or hospital admission in the 6 months prior to the index ED visit, and 2 or more chronic conditions.13 By the end of the study time period, 17% of all ED visits met these eligibility criteria. Exclusion criteria included current enrollment or previous refusal to participate in DISPO ED, residence in a nursing home, and death on date of the index ED visit. 

Data Sources

We used VHA administrative data files, including the Vital Status Mini File,14 enrollment tables from the VHA’s Assistant Deputy Under Secretary for Health for Policy and Planning,15 Medical SAS datasets,16 and additional domains from the Corporate Data Warehouse. 


Primary outcome: repeat VHA ED visit within 90 days. We determined ED use within 90 days of the index visit through administrative stop codes for ED care at US Veterans Affairs medical centers. We determined outpatient ED encounters by using administrative codes for VHA ED visits and VHA inpatient care administrative datasets. 

Key predictors (CAN score). We extracted the CAN score predicting the percentile of risk of hospital admission in the 90 days closest to the index ED visit date and dichotomized CAN scores using the median split (<90 or ≥90). For example, a CAN score of 90 is associated with an average observed hospitalization rate (≤90 days) of 14% compared with an average of 2.7% in the general VHA population.17

ED Super User. Veterans with more than 4 ED visits to the DVAMC within 6 months (including the index ED visit) were categorized as Super Users, based on prior studies and clinical experience.13

Covariates (sociodemographics). Demographics included race, age, marital status, and gender. To indicate economic status, we determined whether the veteran was exempted from co-payments due to limited financial means and had unstable housing within the 12 months prior to the index ED visit.1

Chronic conditions. We used diagnosis codes associated with encounters in the year prior to the index ED visit to identify anemia, congestive heart failure, chronic lung disease, chronic renal failure, diabetes, hypertension, ischemic heart disease (IHD),  peripheral vascular disease, and mental health conditions, including anxiety disorder, depressive disorder, posttraumatic stress disorder (PTSD), and substance abuse disorder, in accordance with the VHA definition of chronic conditions per the VHA Support Service Center Chronic Disease Registry Development Rules.13,18

Medical complexity (Quan Charlson Comorbidity Index). The Quan Charlson Comorbidity Index predicts mortality within 12 months using 17 comorbidities based on the original Charlson Comorbidity Index,19 but using updated weights identified by Schneeweiss et al.20

Outpatient utilization in year prior to index ED visit. We counted the number of VHA primary care, outpatient specialty services, and mental health clinic encounters.

Statistical Analysis

We first compared high-risk classification by CAN score of ≥90 and Super User status, using McNemar’s test. Second, we examined differences in demographics, chronic conditions, and utilization in the year prior to the index ED visit across the 4 classification groups: high-risk by both (CAN score ≥90 and identified Super User), high-risk by CAN score only (CAN score ≥90 and non–Super User status), high-risk by Super User status only (CAN Score <90 and identified Super User), or not considered high risk by both (CAN score <90 and non–Super User). For categorical variables, we used χ2 analysis. Analysis of variance was used for continuous variables and Poisson regressions for count variables. Finally, we compared repeat ED visits within 90 days (yes/no) for these 4 groups, examining CAN score and Super User status in logistic models, adjusting for the aforementioned demographic, economic, comorbidity, and prior healthcare use covariates.

Study Cohort Characteristics

Fifty percent of participants were African American, and 46% were white. The majority (81%) were exempt from co-payments due to financial need. Moreover, 80% had hypertension, 50% had diabetes, and 28% had IHD. Additionally, 66% had 1 or more mental health conditions: 31% were diagnosed with PTSD, 41% with depression, and 19% with anxiety. This cohort demonstrated a high level of engagement with the VHA in the year prior to their index ED visit, with 49% having had 1 or more mental health outpatient encounters and 93% having had 1 or more outpatient specialty service encounters. Veterans with mental health, outpatient specialty, or primary care use in the year prior to the index ED visit had an average of 11 mental health, 9 specialty, and 5 primary care outpatient clinic encounters for the year, respectively (eAppendix A [eAppendices available at]).

Risk Classification by CAN Score Versus Super User Status

Using McNemar’s test, the rate of classification between the 2 methods was discrepant (P <.0001) (Table 1). Based on a CAN score of ≥90, the rate of high-risk classification was 52.8% (n = 814) versus 19.7% (n = 304) for ED Super Users. Of those with a CAN score of 90 or above (n = 814), 76% were not identified as Super Users. Of Super Users (n = 304), 34.9% had a CAN score greater than 90. The 2 methods identified different sets of patients as high risk. 

Characteristics and Utilization Outcomes by Risk Classification of CAN Score and Super User Status

In general, there were few differences in characteristics across the 4 groups based on cross-classification of CAN score of ≥90 and Super User status (eAppendix A). Significant differences included gender (P <.001), primary care utilization in the year prior to the index ED visit (P <.001), specialty care and mental health care utilization in the year prior to the index ED visit (P <.001), and number of ED visits in the year prior to the index ED visit (P <.05). Patients who were identified as Super Users and had a CAN score below 90 had the highest number of primary care visits in the year prior, and those identified as high risk by both methods had the highest number of mental health visits. There were no significant differences in proportion with chronic conditions, with the exception of anxiety (P <.05).

Repeat ED Visits

Overall, 63% (n = 973) of the cohort had 1 or more repeat ED visits within 90 days after the index ED visit (mean = 1.7 repeat ED visits within the observation period); 90% of which were outpatient ED visits. When examining only repeat outpatient ED visits, 59% (n = 906) of the cohort had more than 1 repeat encounter within 90 days. Veterans with a repeat outpatient ED visit had an average of 1.6 repeat ED visits. Six percent (n = 92) of patients died within 90 days of their index ED visit.

There were no statistically significant differences in the proportion of patients with 1 or more repeat ED visits within 90 days of the index encounter, the number of repeat ED visits, or outpatient ED visits across the 4 risk groups (eAppendix B).

Risk classification of the 4 groups was not associated with repeat ED visits within 90 days of the index encounter (P = .28). Adjustment for covariates yielded similar results. When limiting the outcomes to outpatient ED visits within 90 days, we found similar results (Table 2).


In our cohort of VHA ED patients with multimorbidity (defined as 2 or more chronic conditions) and history of a prior ED visit or hospitalization, nearly two-thirds had at least 1 repeat ED visit within 90 days. We found that the CAN score and ED Super User status identified different groups of patients. When we examined 4 risk stratification groups cross classifying by CAN score and Super User status, we found no association between the risk classification groups and repeat ED visits within 90 days of the index date.

Our findings of high repeat ED use in the cohort overall are consistent with previous studies of older veterans, which found that high rates of chronic conditions and prior ED and hospital use were independent predictors of repeat ED use.6,21,22 Compared with the general VHA ED population, of which 15% of patients had a repeat ED visit within 30 days, our cohort had much higher repeat ED utilization.1 Considering these past studies, our cohort was more racially diverse6,23,24 and had more mental health conditions.25 

The fact that these 2 methods of risk stratification did not improve prediction of ED returns in this already high-risk population has important clinical and research implications. First, new applications of existing risk prediction tools should be validated before being put into practice. CAN scores have been available to primary care providers throughout the VA system; however, more data about their clinical utility outside of recognizing patients at high risk for hospitalization and mortality are needed before they are repurposed. Second, although the CAN model, along with other EHR-based methods of risk stratification, incorporates diverse information on patient demographics, medical conditions, and previous utilization, it excludes potentially important data, such as socioeconomic, cultural, and other contextual factors that often play significant roles.26 Third, considering the CAN score is a comprehensive model of health status predictors, improvements in prediction may be achieved through the use of alternative models more frequently seen in other disciplines, such as models based on machine learning techniques.27-30

Although we did not identify subgroups of patients at higher risk of ED returns based on CAN scores, ED Super User status, or a combination of these variables, an examination of the cohort characteristics reveals possibilities for future study into clinical populations of interest and potential ways to improve care. First, we observed high engagement with VHA services, with an average of 5 primary care clinic encounters in the year prior to the index ED visit. Previously, lack of access and low engagement with PCPs and specialists have been highlighted as associated with increased ED use.31-35 Our results suggest that high repeat ED visit rates may not be due exclusively to access barriers; other issues, such as inadequate care coordination, may also play a role.36 If this is the case, increased use of strategies, such as telehealth, may be essential to reducing repeat ED visits. Moreover, the prevalence of mental health diagnoses in our cohort (49%; mean = 11 mental health outpatient encounters in the year prior) was much higher than other studies examining recurring ED use25 and may have contributed to the higher rates of repeat ED visits. Future research should consider the high rates of mental health conditions explicitly. It is notable that the VHA has more extensive and available mental health services compared with many non-VHA systems,36 and interventions directed at non-VHA patients may need to surmount additional barriers to access for mental health services.


There are several limitations of this study. First, we identified VHA ED utilization and diagnoses of interest to veterans using VHA administrative claims data, which are subject to coding bias, errors in record keeping, and delayed records of utilization. Second, the single-site cohort potentially limited generalization. Third, we had limited information about other potentially relevant variables related to ED use, such as socioeconomic status data. Fourth, no gold-standard definition of the term "Super User" exists. We examined ED visits over a 9-month period, which does not address the issue of seasonality. We also only examined repeat ED visits at a single time point (within 90 days); findings may have been somewhat different with a longer time horizon (ie, 6, 9, and 12 months). To address these limitations, we extracted data after allowing sufficient time for records to be updated and relied on clinical expertise from senior researchers regarding best practices to determine diagnoses using claims data. We also relied on the literature, clinical expertise, and prior work on the distribution of ED utilization to define Super User status.12 As an exploratory analysis of associations using secondary data, there was no power calculation for this study. However, the percentages of repeat ED use across the different categorizations were similar and generally above 60%. Thus, there were no indications of significant differences we were not powered to detect; the narrow widths of the confidence intervals are reasonable from the logistic models.

In conclusion, among DVAMC users with multimorbidity and more than 2 prior ED visits or hospitalizations, repeat ED use within 90 days was very high. Applying 2 methods of risk stratification in this population identified discrepant groups of patients, and classification of risk by these 2 measures was not associated with repeat ED use within 90 days. Identifying clinically relevant subgroups is important for future interventions to improve care and provide high-value services for high-risk groups defined by multimorbidity and utilization.


The study was funded by the Department of Veterans Affairs, Health Services Research and Development Service (IIR 12-052) and was also supported by the Durham VA Center for Health Services Research in Primary Care and Geriatrics Research, Education and Clinical Center. Dr Weinberger is supported by the Research Career Scientist Program (RCS 91-408). The VA Office of Academic Affiliations provided fellowship support for Dr Duan-Porter (No. TPP 21-022). The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official views of the Department of Veterans Affairs.

This article has been corrected in Am J Manag Care. 2019;25(3):140.
Print | AJMC Printing...