Edwin S. Wong, PhD

 The Veterans Affairs (VA) Health Care System is among the largest integrated health systems in the United States. Many VA enrollees are dual users of Medicare, and little research has examined methods to most accurately predict which veterans will be mostly reliant on VA services in the future. This study examined whether machine learning methods can better predict future reliance on VA primary care compared with traditional statistical methods.

 Observational study of 83,143 VA patients dually enrolled in fee-for-service Medicare using VA and Medicare administrative databases and the 2012 Survey of Healthcare Experiences of Patients.

 The primary outcome was a dichotomous measure denoting whether patients obtained more than 50% of all primary care visits (VA + Medicare) from VA. We compared the performance of 6 candidate models—logistic regression, elastic net regression, decision trees, random forest, gradient boosting machine, and neural network&mdash;in predicting 2013 reliance as a function of 61 patient characteristics observed in 2012. We measured performance using the cross-validated area under the receiver operating characteristic (AUROC) metric.

 Overall, 72.9% and 74.5% of veterans were mostly VA reliant in 2012 and 2013, respectively. All models had similar average AUROCs, ranging from 0.873 to 0.892. The best-performing model used gradient boosting machine, which exhibited modestly higher AUROC and similar variance compared with standard logistic regression.

 The modest gains in performance from the best-performing model, gradient boosting machine, are unlikely to outweigh inherent drawbacks, including computational complexity and limited interpretability compared with traditional logistic regression.

Reliance on the Veterans Affairs Health Care System for outpatient services is a key metric for informing resource allocation and current operational priorities, including efforts to compete for veteran demand amid increasing healthcare options.

The application of each of 6 candidate models for predicting future reliance exhibited very good performance based on previous thresholds.

Each of the 5 machine learning models considered did not outperform, or modestly outperformed, a traditional regression model.

Modest gains in performance of machine learning models are unlikely to outweigh practical limitations including computational complexity and difficulty in interpreting model composition.

The Veterans Affairs (VA) Health Care System is among the largest integrated healthcare systems in the United States. The VA provides comprehensive health services to eligible veterans, and most of the 8.3 million VA enrollees in 2017 were exempt from co-payments or other cost sharing.

 VA enrollees are unique because, in addition to access to VA health services, more than 80% are dually enrolled in at least 1 other source of private or public insurance. Although most users of VA are exempt from cost sharing, many veterans 65 years or older supplement VA care with Medicare services. Extensive research has examined factors associated with greater dual use of VA and Medicare outpatient services. Dual VA-Medicare enrollees who were more reliant on VA outpatient care were younger and unmarried, lived closer to VA facilities, and were exempt from VA co-payments.

Previous research examining determinants of VA utilization and dual use has largely been descriptive and has not explored approaches to generate accurate predictions. More recent research outside VA has applied newer machine learning algorithms, demonstrating promise in producing predictions of future health outcomes, utilization, and costs with greater accuracy compared with more traditional approaches.

 Notably, several studies have examined prediction of costs within the context of improving risk adjustment, finding that supervised learning methods outperform such traditional methods as parametric regression.

 Machine learning models are being increasingly applied within VA but have not been routinely used for the purpose of predicting reliance on VA health services. These models often have practical challenges, including greater computational complexity and lack of interpretability relative to traditional statistical models. Thus, it is important to understand trade-offs between potential improvements in model performance and these practical challenges.

Improving the ability to correctly predict patients’ expected utilization and costs is important for at least 2 reasons. First, more accurate predictions can help determine the adequacy of budget and staffing resources needed to provide care for a patient population. This is particularly important for the VA, which is funded annually through a budget appropriation from Congress. This appropriation is determined, in part, from the Enrollee Health Care Projection Model, which includes veterans’ expected reliance on the VA for care as a key input.

 At the individual clinic level, predictions of expected VA primary care reliance are helpful in determining staffing distributions, including ancillary services such as social work and integrated behavioral health that are needed by more frequent users of VA.

More accurate predictions of VA reliance can also help better identify veterans expected to use outpatient services outside the VA. This is significant in light of VA’s current emphasis on competing for veterans as customers given increasing healthcare options available to VA enrollees.

 In addition, accurate reliance predictions may inform care coordination efforts within VA’s patient-centered medical home.

 Receipt of fragmented care across health systems is often associated with poor clinical outcomes and increased hospitalizations.

 As current policies continue to support veterans’ access to care outside the VA, accurate predictions of reliance will help identify patients who need further support to mediate the risks of care fragmentation. To address these policy-planning priorities, we examined whether supervised machine learning methods could improve the ability to predict future reliance on VA primary care among VA-Medicare dual enrollees.

The primary data source was the VA Corporate Data Warehouse (CDW), which houses information on utilization of VA health services, demographic data, and 

International Classification of Diseases, Ninth Revision

 We linked CDW data with Medicare enrollment and claims data to ascertain enrollment in fee-for-service Medicare and utilization. We supplemented VA and Medicare administrative data with survey responses capturing patient experiences with VA outpatient care, as measured using the 2012 VA Survey of Healthcare Experiences of Patients (SHEP) Outpatient Module, which is adapted from the Consumer Assessment of Healthcare Providers and Systems.

 SHEP is managed by the VA Office of Reporting, Analytics, Performance, Improvement and Deployment, which routinely administers the survey to a random sample of veterans who obtained outpatient care at VA outpatient facilities. Finally, we obtained measures of local area health resources in veterans’ residence counties, as reported in the Area Health Resource File.

Using data from the 2012 outpatient SHEP, we identified 222,072 respondents enrolled in VA, of whom 90,819 were also continuously enrolled in fee-for-service Medicare in fiscal year (FY) 2012 and FY 2013. Consistent with prior studies,

 we examined veterans dually enrolled in VA and fee-for-service Medicare because we were unable to capture utilization from other non-VA sources. We then excluded 7182 veterans with no primary care utilization in FY 2012 or FY 2013 whose reliance, by definition, was undefined. After excluding 494 veterans with missing covariates, the final study sample consisted of 83,143 dual VA—fee-for-service Medicare enrollees.

Our primary outcome was veterans’ reliance on VA primary care in FY 2013. We measured reliance as the number of primary care visits received in VA divided by total primary care visits (VA + Medicare). We identified primary care visits in VA and Medicare administrative data through the identification of records in which the provider of care was classified in a specialty considered primary care, including general internal medicine, family medicine, and geriatrics.

 We further required records to contain at least 1 primary care evaluation and management Current Procedural Terminology code.

 We dichotomized the continuous reliance measure to indicate whether veterans were mostly reliant on VA primary care (ie, VA reliance >0.5). The rationale for this binary definition is that prior research found that most VA enrollees were completely reliant on 1 system.

We predicted reliance on VA primary care as a function of 61 variables measured in FY 2012, the 12 months prior to the assessment of reliance. The predictor variables encompassed 5 broad categories routinely available to researchers examining VA utilization using administrative data and informed by prior studies examining determinants of VA utilization: patient demographics, access to VA care, physical and mental comorbidities, characteristics of veterans’ residence areas, and experiences with VA care (

 Access to care included several administrative measures, as conceptualized in prior research.

 Comorbidities included 20 indicator variables denoting the presence of conditions in FY 2012, as specified by the validated Gagne et al index.

 Patient experience variables included 23 measures from survey data, elicited on a 4-point Likert scale. These variables included measures capturing perceptions of the availability of VA outpatient care, assessment of VA facilities, and satisfaction with providers and overall VA care. All patient experience measures were defined as binary variables denoting whether an experience dimension was rated in the top 2 categories (eg, always or usually able to obtain immediate care from VA when needed compared with sometimes or never).

Assessing Candidate Machine Learning Algorithms

We compared the performance of several candidate supervised machine learning algorithms in classifying veterans’ future reliance on VA primary care. First, we applied standard logistic regression, which is a traditional approach using parametric regression to model binary outcomes. Next, we applied elastic net regression, which is a regularization approach that combines penalty terms for (1) shrinking regression coefficients to 0, as done in LASSO regression, and (2) shrinking coefficients toward 0, as done in Ridge regression. Third, we considered decision trees, which repeatedly partition the predictor space into subspaces producing the biggest performance improvement. Fourth, we applied random forests, which extend decision trees, by constructing multiple regression trees using bootstrapped training sets, splitting over a subset of predictors, and averaging predictions across trees. Fifth, we examined a gradient boosting machine, which sequentially constructs a regression tree using information from previously grown trees and generates an overall prediction using a weighted average of predictions across trees. Finally, we applied an artificial neural network, which involves constructing a directed graph composed of a network of nodes connected using weights that are updated using predefined rules.

To estimate the performance of the 6 algorithms, we conducted 10-fold cross-validation repeated 3 times, which is a common approach to protect against model overfit.

 We prespecified a grid of tuning parameters specific to the algorithm that controlled the configuration of a model (eg, maximum number of splits for tree-based models). For each combination of tuning parameters, we estimated model parameters and calculated performance metrics for each of the 30 training samples. For each algorithm, we identified the set of tuning parameters with the highest mean performance metric. We compared model performance using the area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity.

 All models applied sampling weights to account for nonresponse of veterans who were offered the 2012 SHEP. We conducted statistical analyses using RStudio version 0.99.891 (RStudio Inc; Boston, Massachusetts).

Implementation of logistic regression, elastic net regression, decision trees, random forests, gradient boosting machine, and neutral networks applied the glm,

 packages, respectively. Assessment of candidate models was facilitated using the caret package.

The characteristics of dual VA-Medicare enrollees were similar to those found in prior studies.

 Specifically, the mean age was 70.7 years, and the majority of patients were male (96.6%), white (78.3%), married (65.7%), and exempt from co-payments for VA health services (76.3%) (Table 1). Compared with patients mostly reliant on VA care (ie, receiving >50% of primary care from the VA in FY 2012), Medicare-reliant patients were older and more likely to be white, married, or reside in urban areas. Medicare-reliant patients were also less likely to be reliant on VA services in the prior year, 2011. Differences between groups were observed for several physical and mental comorbidities, with Medicare-reliant patients having a lower prevalence of heart failure, chronic obstructive pulmonary disease, and psychosis. Also, Medicare-reliant patients reported substantially worse experiences with VA services, including a lower proportion perceiving an ability to access immediate care or specialty care in a timely manner.

The mean (SD) AUROC estimated using logistic regression across 30 validation samples was 0.890 (0.004) (

). Regularized regression using elastic net produced similar predictive performance but exhibited greater variance (AUROC, 0.891; SD = 0.005). Of the 3 tree-based algorithms, decision trees performed the worst (AUROC, 0.873; SD = 0.005), followed by random forest (AUROC, 0.889; SD = 0.003) and gradient boosting machine (AUROC, 0.892; SD = 0.004). The latter exhibited the best performance of all algorithms considered.

Performance was also similar across the 6 models using other evaluation metrics. Mean (SD) specificity ranged from 0.724 (0.007) using decision trees to 0.759 (0.008) for gradient boosting machine. Specificity was modestly lower for logistic regression (mean [SD] = 0.752 [0.007]) compared with the best-performing model. Mean (SD) sensitivity ranged from 0.916 (0.003) for decision tree to 0.922 (0.002) for random forest. Similarly, sensitivity for logistic regression (mean [SD] = 0.920 [0.002]) was only modestly lower compared with the best-performing machine learning model.

Prior study findings have established that dual use of VA and non-VA services is common among VA enrollees. Most studies, however, are solely descriptive and do not have the objective of accurate prediction of future VA reliance. In this study, we explored whether several candidate machine learning models could better predict which patients would be highly reliant on VA primary care services compared with logistic regression, which is commonly used for prediction in health services research. The predictive model using logistic regression exhibited very good performance, as defined using AUROC thresholds from a prior study. However, average performance, as measured by AUROC, was similar across all 6 models considered and varied by less than 2 percentage points. Gradient boosting machine exhibited the best performance of all 6 models considered. However, the predictive performance was less than 1 percentage point better and exhibited modestly greater variance compared with logistic regression. Similarities in model performance were observed for 2 alternative metrics, specificity and sensitivity, with the best-performing models only modestly outperforming logistic regression.

Evidence on how to more accurately predict VA reliance is important for policy planning because the VA is publicly funded through an annual Congressional appropriation, which is informed by budget projection models that include veterans’ expected reliance on VA healthcare as an input. Research in this study is also consistent with the explicit operational goal of focusing resources efficiently within the VA FY 2018-2024 Strategic Plan.

 Specifically, future VA efforts will use VA funding requests to “eliminate spending without sacrificing the outcomes for veterans.” Clinically, reliance prediction models have utility by identifying veterans who are at risk of receiving fragmented care from multiple health systems. For these veterans, creating services to improve care coordination can avoid duplication of services, reduce waste, and improve patient safety by mitigating the loss of clinical information needed by providers.

Stakeholders determining the best model to apply when predicting future VA reliance should consider a number of factors other than overall performance. These considerations include the model assumptions, computational complexity, variance in model performance, and intended use. Notably, more complex models such as random forest and gradient boosting machine tend to exhibit greater performance, on average, but may overfit data, resulting in higher variance and less reliable predictions when applying models in the real world. In addition, more complex models are generally more computationally intensive,

require more time to train, and may be less desirable if stakeholders require that models be frequently updated to incorporate newly collected data. If interpretability is an important feature, more complex models may be less desirable despite better overall performance because model parameters that reflect effect of individual predictors, analogous to regression coefficients, are often not available. For the prediction of future VA primary care reliance using the 61 variables considered in this study, the practical limitations of more complex machine learning models likely outweigh the modest gains in predictive performance relative to traditional logistic regression, which achieved very good performance. Because model performance varies with the specific set of predictor variables chosen, the optimal prediction algorithm may differ for an alternative set of data.

This study has a number of limitations. First, our study sample consisted of a sample of VA-Medicare dual enrollees who responded to a routinely administered VA patient experience survey to capture a wider set of predictor variables not available in VA administrative data. Because these patients were recent users of the VA, and veterans remain enrolled until death regardless of utilization, the study sample may not generalize to the population of all VA enrollees. Second, we were able to measure utilization of primary care services from VA and Medicare using a unique data-sharing agreement available to VA researchers. However, we were unable to capture utilization of primary care services through other payers. Third, data used in this study measured VA primary care reliance prior to the expansion of community care options in 2014.

 To date, the uptake of these community care options for primary care services among VA-Medicare dual enrollees is unknown.

Knowledge of reliance on VA services is important for policy planning and care delivery, but there has been limited research on improving prediction of this important outcome. We examined several candidate methods for predicting reliance as a function of a comprehensive set of patient variables. We found that more complex machine learning methods exhibited modest improvements in predictive performance compared with traditional logistic regression. These gains in performance are unlikely to outweigh other practical limitations of more complex machine learning models that are more computationally intensive and difficult to interpret.

 Center for Veteran-Centered and Value-Driven Care, VA Puget Sound Health Care System (ESW, LS, AR), Seattle, WA; Department of Health Services (ESW) and Division of General Internal Medicine, Department of Medicine (LS, AR), University of Washington, Seattle, WA.

 Dr Wong was supported by a Veterans Affairs Health Services Research and Development Career Development Award (CDA 13-024). The views expressed are those of the authors and do not necessarily reflect the views of the Department of Veterans Affairs and the University of Washington.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (ESW, AR); acquisition of data (ESW); analysis and interpretation of data (ESW, LS, AR); drafting of the manuscript (ESW, LS, AR); critical revision of the manuscript for important intellectual content (LS, AR); statistical analysis (ESW); and obtaining funding (ESW).

 Edwin S. Wong, PhD, Center for Veteran-Centered and Value-Driven Care, VA Puget Sound Health Care System, 1660 S Columbian Way, HSR&D MS-152, Seattle, WA 98108. Email: edwin.wong@va.gov.

1. Huang G, Kim S, Muz B, Gasper J. 2017 survey of veteran enrollees’ health and use of health care. US Department of Veterans Affairs website. va.gov/HEALTHPOLICYPLANNING/SOE2017/VA_Enrollees_Report_Data_Findings_Report2.pdf. Published April 2018. Accessed June 1, 2019.

2. Hynes DM, Koelling K, Stroupe K, et al. Veterans’ access to and use of Medicare and Veterans Affairs health care. 

. 2007;45(3):214-223. doi: 10.1097/01.mlr.0000244657.90074.b7.

3. Liu CF, Manning WG, Burgess JF Jr, et al. Reliance on Veterans Affairs outpatient care by Medicare-eligible veterans. 

. 2011;49(10):911-917. doi: 10.1097/MLR.0b013e31822396c5.

4. Petersen LA, Byrne MM, Daw CN, Hasche J, Reis B, Pietz K. Relationship between clinical conditions and use of Veterans Affairs health care among Medicare-enrolled veterans. 

. 2010;45(3):762-791. doi: 10.1111/j.1475-6773.2010.01107.x.

5. Morid MA, Kawamoto K, Ault T, Dorius J, Abdelrahman S. Supervised learning methods for predicting healthcare costs: systematic literature review and empirical evaluation. 

6. Shrestha A, Bergquist S, Montz E, Rose S. Mental health risk adjustment with clinical categories and machine learning. 

. 2018;53(suppl 1):3189-3206. doi: 10.1111/1475-6773.12818.

7. Park S, Basu A. Alternative evaluation metrics for risk adjustment methods. 

. 2018;27(6):984-1010. doi: 10.1002/hec.3657.

8. Rose S. A machine learning framework for plan payment risk adjustment. 

. 2016;51(6):2358-2374. doi: 10.1111/1475-6773.12464.

Articles

Machine learning models, used to predict future use of primary care services from the Veterans Affairs (VA) Health Care System, did not outperform traditional regression models.

Does Machine Learning Improve Prediction of VA Primary Care Reliance?

 Many healthcare systems use prediction models to estimate and manage patient-level probability of hospitalization. Patients identified as high-risk at one point in time may not, however, remain high-risk. We aimed to describe subgroups of patients with distinct longitudinal risk score patterns to inform interventions tailored to patients’ needs.

 Using a previously validated prediction algorithm, we identified a cohort of 258,759 patients enrolled in the Veterans Health Administration (VHA) who were in the top 5% of risk for hospitalization within 90 days. During each of the following 24 months, patients were placed in 1 of 6 categories: death, hospitalized, no VHA care, persistently high-risk for hospitalization (≥10% probability), initially high-risk then persistently low-risk (<10% probability), and intermittently high-risk. We used multivariable logistic regression to identify characteristics predictive of being persistently high-risk through the last study month.

 After 2 years, 17.7% had died, 13.8% had remained persistently high-risk for hospitalization, 41.5% had become persistently low-risk, and 19.9% were intermittently high-risk. Predictors of being persistently high-risk included urban residence, chronic medical comorbidities, auditory and visual impairment, chronic pain, any cancer diagnosis, and social instability.

 Few patients who were high-risk for hospitalization at baseline remained so. Nonrandomized evaluations of interventions that identify patients based on a single high-risk score may spuriously appear to have positive effects. Clinical interventions may need to focus on individuals who are persistently high-risk.

Many patients identified as high-risk for near-term hospitalizations using a risk prediction model at baseline did not remain high-risk over a 2-year period.

After 2 years, just 14% remained persistently high-risk for hospitalization. Of the remainder, 20% were intermittently high-risk, 42% became persistently low-risk, and 18% had died.

Predictors of being persistently high-risk included age, gender, urban residence, chronic pain, any cancer diagnosis, chronic medical comorbidities, and social instability.

Nonrandomized evaluations of interventions that target patients based on a single high-risk score may spuriously appear to improve patient outcomes due to natural decreasing risk over time.

The development of advanced healthcare analytics allows large healthcare organizations to use risk stratification as a population management strategy to proactively identify patients at high risk for poor health outcomes. These high-risk patients may benefit from customized interventions with the goals of improving outcomes and providing healthcare more efficiently.

 Risk stratification or risk prediction tools are often based on historical administrative claims and utilization data.

 However, the stability of risk estimates over time is uncertain. Programs that target patients identified as being high-risk at one point in time do not take into account the dynamic nature of changing risk over time; therefore, such programs may vary in effectiveness across patients.

 Different clinical interventions may be indicated for patients who are persistently versus temporarily high-risk.

Studies have supported the notion that many high-cost, high-needs patients are just temporarily high-risk.

 Documented predictors of hospital and emergency department (ED) utilization include schizophrenia, homelessness, opioid prescriptions, depression, substance use, social isolation, and heart failure.

 However, little information exists to characterize patients who remain persistently high-risk for hospitalization. Prediction of persistent high risk over time may enhance risk stratification as a population management strategy and inform efforts to target and tailor resource-intensive clinical interventions.

The Veterans Health Administration (VHA) incorporated risk stratification measures into practice nationally in 2012. It reports weekly estimates of the probability of patient hospitalization in the next 90 days for the entire primary care patient population for use by primary care teams to inform clinical decision making.

 We aimed to inform population management programs for high-risk patients

 by following a cohort of high-risk patients over a 2-year period. Our objectives were to identify and describe groups of high-risk patients with distinct longitudinal risk score patterns and identify baseline characteristics that predicted which patients were likely to remain high-risk over time.

Data for this study were obtained from the VHA Corporate Data Warehouse, which included patient demographics, clinical diagnoses, admission and discharge status, patient risk scores, and utilization records.

 We also obtained data from the VHA Vital Status file to ascertain patients’ dates of death.

We identified a national cohort of 258,759 patients who had a 10% or greater absolute probability of 90-day hospitalization on our index date (November 23, 2012), were assigned to a VHA primary care provider, and were not hospitalized in the VHA on the index date, because risk scores are not generated for hospitalized patients. We followed patients for 2 years after the index date.

We obtained patients’ monthly risk using the predicted probability of hospitalization in the next 90 days in the Care Assessment Need (CAN) model, a validated risk prediction algorithm.

]) uses demographics, utilization of VHA health services, comorbidities, prescribed medications, vital signs, and veteran-specific variables.

Risk scores were generated on a weekly basis for all patients in VHA primary care. Patient-month measures of risk were calculated as the mean of weekly probabilities for that calendar month, and missing values occurred when veterans were hospitalized (including acute care, residential treatment facility admission, nursing home admission, or hospice admission) or dropped out of VHA care.

We defined patients as “high-risk” if the probability of 90-day hospitalization was 10% or greater, which corresponds to approximately the top 5% of VHA patients (n = 258,759). Our main dependent variable was a patient-month measure of risk persistence, defined hierarchically (

Patients with at least 1 missing CAN probability in a calendar month were categorized into 1 of the following mutually exclusive outcomes: death, hospitalization in VHA, or dropped out of VHA care (no VHA encounters in previous 24 months). By definition, patients can move into and out of the “hospitalization” or “dropped out of care” categories. All patients except those who died were recategorized the following month.

 We categorized patients with nonmissing CAN scores as persistently high-risk, intermittently high-risk, or initially high-risk then persistently low-risk for every month based on all patient-month observations in the study period. Patient-month observations were coded as “persistently high-risk” if hospitalization probability was 10% or greater during the current month and all prior months. “Initially high-risk” was defined as probability less than 10% during all subsequent months. “Intermittently high-risk” was defined as having a combination of high (≥10%) and low (<10%) hospitalization risk across the months and not meeting the criteria for persistently high-risk or initially high-risk. These categorizations ignore the patient-months designated as “hospitalization” or “dropped out of care.” By definition, patients can only move out of the persistently high-risk group, patients can only move into the initially high-risk group, and patients may move into or out of the intermittently high-risk group.

We examined patient characteristics measured at baseline: demographics (age, gender, marital status, ethnicity, urban vs rural residence), medical and psychiatric comorbidities, any cancer diagnosis, and markers of social instability.

 Markers of social instability included discharges against medical advice

; number of zip code changes in the year prior

) diagnosis of nonadherence to medical treatment; and homelessness. Homelessness was measured by use of housing services

 diagnosis codes indicating lack of housing. 

 diagnoses for comorbidities were adapted from previously published research

 and from the VHA Health Economics Resource Center

). Numbers of hospitalizations and outpatient visits and no-show rates were calculated from the index date over the subsequent 2 years. No-shows were recorded for each instance in which a patient did not arrive to a scheduled appointment.

For outpatient visits during the follow-up period, we counted VHA encounters that occurred face-to-face or by telephone with any healthcare provider (eg, physician, nurse, psychologist). We categorized outpatient visits based on the specific type of service (ie, primary care, mental health, palliative care, ED).

 Mental health visits included primary care—mental health integration encounters.

 Inpatient admissions were categorized into medical, surgical, and psychiatric admissions.

Our dependent variable in bivariable and multivariable analyses was the patient’s risk category (ie, persistently high, intermittently high, or initially high) in the final study month (n = 194,597). We used analysis of variance to examine whether there were differences in means among the continuous variables (inpatient/outpatient utilization, ED visits, and zip code changes) across the high-risk categories. In bivariable analyses, we performed ordered logit regression to estimate a single equation for each independent variable over the 3 ordered levels of the dependent variable: persistently, intermittently, and initially high-risk. Proportional odds ratios (ORs) were generated from the ordered logit models to estimate the change in odds in each predictor for patients at (1) persistently high-risk versus intermittently high-risk and (2) intermittently high-risk versus initially high-risk.

We performed multivariable logistic regression to predict being persistently high-risk versus the other 2 groups after adjusting for independent variables found to be statistically significant (

 <.05) from the ordered logit regression analyses described previously. Model covariates included age, gender, and race/ethnicity, and we excluded variables used to calculate the CAN probabilities (eAppendix Table 1).

For continuous independent variables that are not normally distributed (ie, number of discharges against medical advice, palliative care encounters, mental health visits, and zip code changes), we applied a log (base 10) transformation to create a less skewed distribution. As a result, one would need to have a 10-fold increase in the independent variables to achieve the ORs indicated for the transformed variables.

Because the ordered logit model assumes proportional odds, we also estimated a generalized ordered logistic regression in sensitivity analysis. This model estimates ORs for variables that violate the proportional odds assumption separately for each comparison.

 OR estimates from the generalized ordered logit model were of similar size and significance as those from the ordered logit model (

). In sensitivity analyses, we also used a multinomial logit model to compare patients who were persistently high-risk with those who were intermittently high-risk and to compare those who were persistently high-risk with those who were initially high-risk. Directionality and significance were similar to the ordered logit model (

We conducted stratified analyses by age group (≥65 years and <65 years) to examine whether there were differences in risk patterns between VHA enrollees eligible for Medicare and younger veterans; results were similar for both age groups (

), so we report only the combined results.

Analyses were conducted with SAS version 9.4 (SAS Institute Inc; Cary, North Carolina) and SAS Enterprise Miner.

This evaluation was designed to support VHA operations and quality improvement for the VHA Offices of Primary Care and Clinical Systems Development and Evaluation and was exempt from institutional review board approval or waiver.

 shows the monthly risk status of the high-risk patient cohort over a 24-month period (quantitative results in 

). After 1 year, 28.6% (n = 74,060) were persistently high-risk for hospitalization, 10.3% (n = 26,541) had died, 23.8% (n = 61,556) were initially high-risk for hospitalization, and 30.6% (n = 79,078) were intermittently high-risk for hospitalization. By the end of the 2-year period, just 13.8% (n = 35,770) remained persistently high-risk for hospitalization; 17.7% (n = 45,805) had died, 41.5% (n = 107,473) were initially high-risk, and 19.9% (n = 51,354) were intermittently high-risk, and the remaining 7.1% (n = 18,357) were hospitalized or left VHA care at study end.

Patients who were older than 45 years, male, unmarried, black, and living in an urban area had greater odds of being persistently high-risk for hospitalization (vs intermittently high-risk) and of being intermittently high-risk for hospitalization (vs initially high-risk) (

). Patients with nearly all medical and mental comorbidities had greater odds of being persistently or intermittently high-risk for hospitalization. The largest differences between the risk categories were found with congestive heart failure (OR, 2.60; 95% CI, 2.55-2.65), chronic kidney disease (OR, 2.48; 95% CI, 2.39-2.58), and dementia (OR, 1.96; 95% CI, 1.90-2.00). Patients with social instability, such as discharges against medical advice, number of zip code changes, and diagnoses of nonadherence, had greater odds of being both persistently and intermittently high-risk for hospitalization. Median no-show rates in a given year were similar for all risk categories.

Patients who were persistently high-risk for hospitalization were more likely than those who were intermittently high-risk or initially high-risk to have a VHA hospitalization (89%, 75%, and 41%, respectively) or ED visit (93%, 86%, and 57%, respectively) during the follow-up period (

). Also, persistently high-risk patients had more frequent VHA hospitalizations per year than the initially high-risk group, with a mean (SD) of 1.9 (3.5) hospitalizations per year; most (1.5 [3.1]) of these admissions were medical.

Patients who were persistently high-risk for hospitalization also had higher rates of outpatient utilization: They had a mean (SD) of 83.2 (81.2) total VHA outpatient encounters (telephone and in-person visits) per year, almost double the number of outpatient encounters per year among patients who were only initially high-risk for hospitalization (Table 2). Persistently high-risk patients had a mean (SD) of 10.6 (15.9) encounters in primary care, 18.1 (64.9) encounters in mental health, and 0.1 (1.9) encounters in palliative care per year. In contrast, patients who were only initially high-risk for hospitalization had, on average, 44.4 (58.4) outpatient encounters per year, 6.7 (11.0) in primary care, 9.5 (41.1) in mental health, and 0.02 (0.8) in palliative care. Intermittently high-risk patients had mean utilization in between the persistently high-risk and initially high-risk groups.

In multivariable analyses, we identified 20 statistically significant (

 <.05) predictors of being persistently high-risk for hospitalization, including age, gender, urban residence, visual and hearing impairment, chronic pain (back and neck pain, arthritis, and headache), chronic medical comorbidities (chronic kidney disease, coronary artery disease, congestive heart failure, diabetes, hypertension, and nicotine use), number of visits to palliative care, cancer diagnosis, number of mental health visits, and all markers of social instability (

In this study, we examined the patterns of risk status over time among a population of high-risk patients receiving continuous VHA care. Consistent with findings of prior research on costs and readmissions,

 the majority of these patients did not remain persistently high-risk during 2 years of follow-up; just 29% were persistently high-risk after 1 year, and 14% remained persistently high-risk after 2 years. Almost half (42%) were classified as being persistently low-risk by study end.

These findings may partly explain seemingly positive results from interventions that enroll patients who are high-risk at baseline but that report pre—post findings without a comparison group. The improvement in patient outcomes could, in fact, be due to naturally decreasing risk over time rather than due to the intervention itself. Several randomized studies of care management interventions for high-need, high-cost patients have found no differences in cost and utilization between intervention and usual-care groups.

We found significant and meaningful differences in socio&shy;demographics, clinical comorbidities, and utilization among the 3 trajectory groupings of high-risk patients. The presence of chronic medical or behavioral conditions (particularly heart failure and chronic kidney disease), a nonadherence diagnosis, and higher VHA utilization rates were associated with greater duration of remaining persistently high-risk for hospitalization, which indicates that programs for high-risk patients can focus efforts on patients with these characteristics. Persistently and intermittently high-risk patients used a remarkable amount of VHA services—as many as an average of 63 to 83 outpatient visits and 0.9 to 1.9 admissions per year&mdash;suggesting that they were actively engaged with ambulatory care providers. Some acute care may be unavoidable despite high utilization of outpatient services, especially for patients who rapidly decompensate or cannot be managed effectively with ambulatory care alone.

 Persistently high-risk patients were more likely to live in urban areas, but many (30%) lived in rural areas, indicating that smaller, rural community clinics or virtual care modalities may have an important role to play in mitigating risk for these patients.

 Our results confirm published findings described in other predictive models for high utilization, readmission, and mortality: Age, gender, number of zip code changes, number of discharges against medical advice,

 frequent use of outpatient medical and mental health services,

 significantly predicted long-term persistence of high risk. Other known predictors of high utilization (eg, socioeconomic status, depression, alcohol use disorder, marital status) were not included in our model due to endogeneity. Unlike previous studies, we report that hearing and visual impairment, cancer diagnosis, and palliative care utilization were also predictors of patients being persistently high-risk. Interestingly, unlike prior studies

 that preceded the VHA comprehensive primary care teams for homeless patients,

 our findings show that homeless patients who used housing support services were less likely to be persistently high-risk. It is possible that VHA programs that address housing instability were able to effectively change the duration for which a patient is high-risk.

Developing effective programs for high-risk patients has proved challenging because most randomized trials in this area have not shown improvement in outcomes.

 One possibility is that high-risk patients are so heterogeneous that a one-size-fits-all approach may not be effective. Different clinical interventions may be more suitable for patients who are persistently high-risk versus those who are only initially or intermittently high-risk or for patients with various combinations of factors that predict risk.

 For instance, persistently high-risk patients may benefit from longitudinal primary care programs tailored for specific vulnerable populations (eg, primary care for homeless, homebound, or elderly patients; those receiving dialysis; those with chronic pain or serious mental illness); other high-risk patients may benefit from time-limited care management models that augment primary care, such as collaborative care models for high-needs patients.

 Large healthcare systems may find it helpful to use empiric clustering models to further distinguish subgroups of high-risk patients based on clinical characteristics.

Our findings suggest ways to tailor care management interventions for patients with sustained needs. Although the extent to which a patient’s underlying risk for hospitalization is modifiable remains unclear, evidence-based practices can be implemented to possibly reduce a patient’s risk for hospitalization. For instance, interventions to target housing instability, such as the Housing First model, may reduce a patient’s risk for future hospitalization.

 Healthcare teams who target persistently high-risk patients could include behavioral health specialists familiar with managing treatment nonadherence, holistic approaches to chronic pain, and guideline-concordant care for ambulatory care—sensitive conditions (eg, congestive heart failure, diabetes, hypertension). Comprehensive assessments could identify depression and patient factors underlying nonadherence and social instability in high-risk patients and then prioritize which patients should receive the most intensive interventions to reduce future risk. Future studies could examine whether intervening on predictors of persistent high risk is effective in changing a patient’s risk trajectory.

Measures of socioeconomic status, including income, employment status, and education status, were likely predictors of risk persistence, but they were not included due to lack of availability in VHA administrative data.

 To avoid a problem in tautology, we excluded VHA risk prediction model variables (eg, mental health conditions, cognitive impairment) from our model to predict whether or not patients remain at high risk. However, we do report some clinical characteristics not included in the risk prediction model that contribute to hospitalization risk, such as chronic pain conditions, markers of social instability, and sensory impairments. Lastly, our sample is limited to the VHA population, which includes veterans who are less likely to have childhood-onset medical conditions due to military medical eligibility, are more likely to have psychiatric conditions, and may use services in multiple healthcare settings, not just the VHA.

 We were unable to draw from non-VHA data sources for this analysis to account for healthcare services that were not paid by the VHA. Although the actual percentage of persistently high-risk patients might differ by payer,

 the overall patterns are likely to be similar.

Most patients in a large integrated healthcare system who were high-risk for hospitalization were at substantially lower risk within 2 years.

Which Patients Are Persistently High-Risk for Hospitalization?

 Recent financial penalties for high risk-adjusted chronic obstructive pulmonary disease (COPD) readmissions are causing hospitals to search for ways to reduce COPD readmissions. Although some have advocated for increasing the length of stay (LOS) as a method to decrease readmissions, the association between LOS and readmission is unclear. Our primary objective was to examine the association between LOS and readmission among patients admitted for COPD.

We conducted an observational study of 33,558 veterans admitted to 130 Veterans Affairs hospitals for COPD from October 1, 2008, to September 30, 2011. 

We used multivariable regression to separately examine the associations of patient and hospital LOS with 30-day all-cause readmission. 

At the patient level, compared with short LOS (<3 days), a longer LOS was associated with increased risk for readmission. The adjusted odds ratio was 1.39 (95% confidence interval [CI], 1.18-1.63) for medium LOS (3-4 days) and 2.03 (95% CI, 1.72-2.40) for long LOS (>4 days). On the hospital level, there was no association between LOS and readmission. 

On a patient level, a longer LOS for COPD hospitalizations was associated with higher risk for readmission, which is likely confounded by the severity of the illness. On a hospital level, LOS was not associated with readmission. These findings imply that, independent of other transitional care practices, altering the hospital LOS may not influence the risk of readmission.

In a nationwide study of patients admitted for chronic obstructive pulmonary disease (COPD) to Veterans Affairs hospitals, we examined the association between length of stay (LOS) and readmission on the patient level and the hospital level separately. 

		Patient-level LOS was associated with increased risk of readmission, which was likely due to residual confounding of severity of illness. 

		Hospital-level LOS was not associated with readmission, and there was no association between risk-adjusted readmission and LOS. 

		These findings imply that a strategy of simply keeping patients in the hospital longer is not likely to be an effective approach for reducing the risk of readmissions.

With healthcare costs continuing to rise, health systems are searching for ways to improve value-based care. Recently, CMS included readmissions for chronic obstructive pulmonary disease (COPD) in the Hospital Readmission Reduction Program, in which hospitals with excessive risk-adjusted readmission rates are financially penalized.

 Some have argued that keeping patients in the hospital longer would allow more time for optimal medical management and reduce the risk for readmission.

 However, the relationship between length of stay (LOS) and readmission is still unclear, and it is unknown whether increasing LOS is associated with lower readmission rates.

COPD is one of the most common chronic diseases leading to hospitalization in the United States.

 There are more than 700,000 COPD hospitalizations in the United States each year, accounting for more than $6 billion in healthcare spending

; average LOS for patients admitted with COPD varies widely between hospital systems.

 Furthermore, clinical trials on COPD hospitalizations have shown that early hospital discharge with home healthcare can effectively reduce LOS as well as readmissions in select patients.

 These findings suggest that opportunities may exist to safely improve hospital efficiency by reducing LOS for COPD.

Observational studies examining the relationship between LOS and readmission are frequently confounded by the severity of the illness,

 which makes it difficult to study this association. Patients who are sicker tend to require longer LOS and have higher risk of readmission. Several studies have attempted to overcome these challenges by observing variations in readmission rates corresponding to temporal trends in LOS.

 However, these studies have been unable to account for other temporal trends in clinical practice that may impact the outcome. Randomized trials on LOS could provide additional evidence, but they would be methodologically challenging to design and conduct.

The primary objective of this study was to examine the association between LOS and all-cause 30-day hospital readmission among patients who were admitted to Veterans Affairs (VA) hospitals for COPD. We first examined how individual patient LOS was associated with patient-level readmission risk. To determine the impact of organizational influences on patient outcomes, we then assessed how median hospital LOS was associated with patient-level readmission risk. Lastly, we examined the relationship between hospital-level risk-adjusted readmission rates and risk-adjusted LOS.

We performed an observational study of patients admitted to VA hospitals for COPD during the fiscal years 2009 to 2011 (October 1, 2008, to September 30, 2011). We only included veterans who had least 1 year of VA care prior to index hospitalization. We identified 38,128 unique patients with COPD admissions by a principal 

International Classification of Diseases, Ninth Revision, Clinical Modification

) discharge diagnosis of COPD (491.0, 491.1, 491.2, 491.8, 491.9, 492.0, 492.8, 493.2, and 496), which is a validated approach that has been used in other VA studies.

 We excluded hospitals that admitted fewer than 25 patients for COPD during the study period (n = 11) to ensure stability in hospital-level covariate measures and readmission rates. We also excluded patients who died during hospitalization (n = 518), were discharged to a nursing hospital (n = 683), were transferred to a non-VA hospital (n = 2773), had an LOS of less than 1 day (n = 502), or had an LOS longer than 30 days (n = 83). Patients who died within 30 days of hospital discharge represented a small fraction of the total sample (n = 710), and they were included in the analysis. Repeating the analysis after excluding these patients did not substantially change our findings. Data on readmissions were collected from VA administrative databases and Medicare claims. The VA Puget Sound and VA Connecticut Healthcare Systems institutional review boards approved the study.

The 2 independent variables of interest were: 1) patient LOS and 2) hospital median LOS. Patient LOS was calculated as the difference between the calendar date of admission and discharge. Given the skewness of LOS, we classified patient-level LOS into short (<3 days), medium (3-4 days), and long (>4 days), based on tertiles. Hospital LOS was calculated as the median LOS for all patients with COPD admitted within each hospital and was analyzed separately to determine the impact of organizational variation in LOS. To avoid the assumption that the relationship between hospital LOS and readmission was linear, hospital LOS was also broken into 3 categories for analysis. Given that the majority of patients were admitted to a hospital that had a median LOS of 3 days, we examined the following categories: fewer than 3 days, 3 days, and more than 3 days.

The primary outcome of interest was hospital readmission for any cause within 30 days of discharge from index hospitalization. This outcome is consistent with the focus of the Hospital Readmissions Reduction Program (HRRP) on improving diverse readmissions. For each patient, only the first readmission within 30 days after index hospitalization was included in the analysis. Readmission data was collected from VA electronic health records and Medicare claims.

We controlled for baseline patient characteristics including age, gender, race, marital status, distance to index hospital, Medicare and Medicaid status, co-payment status, number of home zip codes in the year prior to index hospitalization (1 vs >1), and discharge against medical advice in the prior year. To measure severity of illness, we included more than 30 diseases using inpatient and outpatient 

codes from the year prior to index hospitalization based on a previously validated risk score.

 The comorbidities included multiple conditions that frequently affect patients with COPD, such as heart disease, pulmonary vascular disease, and cancer. Each comorbidity was included in the model separately, and we also included an overall measure of comorbidity burden prior to index hospitalization using the Hierarchical Condition Category (HCC).

The number of VA hospitalizations and clinic visits in the prior year were divided into quartiles and included as covariates in the analyses. Information on baseline COPD therapy included number of short-acting beta-agonist (SABA) canisters dispensed and prescriptions (yes/no) for nebulized SABAs, ipratropium, tiotropium, long-acting beta-agonists (LABAs), inhaled corticosteroids, and oral steroids in the year prior to index hospitalization. We did not have information on combined LABAs and inhaled corticosteroid medications. We also included information on whether patients were started on new invasive or noninvasive mechanical ventilation during index hospitalization as a proxy for severity of illness. Finally, to account for differences in LOS and readmission rates over time, we included the fixed effects of the fiscal year of patient discharge in the analyses.

We conducted unadjusted analyses, including χ

 tests, to examine the association between LOS and readmission at the patient level and hospital level. We then estimated separate mixed effects multivariable logistic regression models to examine: 1) patient-level LOS and 2) hospital-level LOS. We calculated all-cause risk-adjusted readmission rates for each hospital using multivariable logistic regression and calculated risk-adjusted LOS using multivariable negative binomial regression. For both risk-adjusted models, we controlled for patient characteristics, included fixed effects for the hospital, and clustered at the hospital level. Risk-adjusted rates were predicted for each hospital, with patient characteristics held constant at national mean levels. We graphed the hospital-level risk-adjusted readmission rate and risk-adjusted LOS against each other and calculated Pearson’s correlation coefficient. 

In a sensitivity analysis, we repeated our regression analyses to examine the association between LOS and all-cause 30-day readmission among the frailest populations in our cohort, including: 1) patients older than 75 years and 2) patients who had the most comorbidities based on the highest HCC quartile. We also repeated our analysis using COPD-specific 30-day readmissions.

We included 33,558 patients who were admitted for COPD at 130 hospitals in our analyses. The mean patient LOS was 3.99 days (standard deviation [SD] = 4.06). There was notable skewness in patient LOS (

). Dividing patient LOS into tertiles yielded 13,194 patients who had an LOS shorter than 3 days; 10,760 patients who had an LOS of 3 to 4 days; and 9604 patients who had an LOS longer than 4 days. There was less skewness in median hospital LOS (

). Dividing hospital LOS into 3 categories yielded 6414 patients at 25 hospitals with a median LOS shorter than 3 days; 20,506 patients at 70 hospitals with a median LOS of 3 days; and 6638 patients at 35 hospitals with a median LOS longer than 3 days.

Patients who had a longer LOS tended to be older, were more likely to be unmarried, were more likely to be white, lived farther from the hospital, had more prior hospitalizations, and had more comorbidities (

). Longer LOS was also associated with more severe markers of COPD, including higher numbers of prior COPD medications and more frequent use of mechanical ventilation during index hospitalization. In unadjusted analyses, patient LOS was associated with increased 30-day readmission (15.9% for <3 days LOS vs 21.5% for >4 days LOS; 

). In contrast, there was no significant association between hospital median LOS and readmission.

In multivariable regression models with patient LOS as the independent variable of interest, longer LOS was associated with increased risk for readmission. Relative to short-LOS patients, the odds of readmission were 1.39 (95% confidence interval [CI], 1.18-1.63) and 2.03 (95% CI, 1.72-2.40) times greater for medium- and long-LOS patients, respectively. In contrast, multivariable logistic regression with hospital LOS as an independent variable did not reveal a significant association with readmission. Relative to patients discharged from short-LOS hospitals, the odds of readmission were 1.12 (95% CI, 0.91-1.38) and 1.12 (95% CI, 0.86-2.04) times greater for patients discharged from medium- and long-LOS hospitals, respectively. Graphing the risk-adjusted readmission rate against risk-adjusted LOS did not reveal a significant association between the 2 variables (Pearson correlation = 0.14; 

Results from sensitivity analyses coincided with our primary results. Among 10,376 patients older than 75 years, we found increased risk of readmission on a patient level (medium LOS: odds ratio [OR], 1.51; 95% CI, 1.15-2.00; long LOS: OR, 2.34; 95% CI, 1.78-3.10), but no significant increased risk on a hospital level (medium LOS: OR, 0.91; 95% CI, 0.64-1.31; long LOS: OR, 1.03; 95% CI, 0.71-1.52). Among 8290 patients in the highest HCC quartile, we also found increased risk of readmission on a patient level (medium LOS: OR, 1.63; 95% CI, 1.23-2.16; long LOS: OR, 1.84; 95% CI, 1.39-2.44), but no significant increased risk on a hospital level (medium LOS: OR, 1.47; 95% CI, 0.97-2.23; long LOS: OR, 1.26; 95% CI, 0.80-1.98). Repeating the analysis using COPD-specific 30-day readmissions also did not reveal substantial differences in our results (data not shown).

This study showed that on a patient level, longer LOS for COPD hospitalizations were significantly associated with a higher risk for readmission, which is likely explained by unmeasured sociodemographic factors and greater severity of illness. Patients who are sicker and have less social support tend to have longer LOS and greater risk for readmission. In our study, patients who had longer LOS tended to be older and unmarried, to live farther from the VA hospital, and to have more comorbidities, prior hospitalizations, prescriptions for COPD medications, and greater use of mechanical ventilation during index hospitalization. Even after controlling for these variables, we still found a significant association between patient LOS and readmission, which most likely reflects residual confounding. After using the median hospital LOS in the model, we found no association between LOS and readmission risk. This was true not only in our general analysis, but also in our sensitivity analysis, which examined the association between LOS and readmission for the frailest patients. Variations in median hospital LOS likely reflect differences in organizational culture and practices that impacted decisions to keep patients in the hospital for longer periods of time. Our findings suggest that a strategy of simply keeping patients in the hospital longer is not likely to be an effective approach for reducing the risk of readmissions.

There is limited research on the association between LOS and hospital readmission. A few studies have examined this relationship using temporal trend analyses.

 They found that although LOS decreased for several conditions in the 1990s and early 2000s, there were no corresponding increases in readmission rates. However, during this same period of time, several changes may have independently reduced readmissions. For example, there was a growing financial focus on reducing excess readmissions that resulted in the CMS implementing the HRRP to financially penalize hospitals with high risk-adjusted readmission rates.

Among patients admitted for chronic obstructive pulmonary disease (COPD) at Veterans Affairs hospitals, hospital-level length of stay was not associated with 30-day readmission.

Edwin S. Wong, PhD

Articles

Does Machine Learning Improve Prediction of VA Primary Care Reliance?

Which Patients Are Persistently High-Risk for Hospitalization?

Association Between Length of Stay and Readmission for COPD

Organizational Structure for Chronic Heart Failure and Chronic Obstructive Pulmonary Disease