ACO Quality Over Time: The MSSP Experience and Opportunities for System-Wide Improvement
William K. Bleser, PhD, MSPH; Robert S. Saunders, PhD; David B. Muhlestein, PhD, JD; Spencer Q. Morrison, BA; Hoangmai H. Pham, MD, MPH; and Mark B. McClellan, MD, PhD
Accountable care organizations (ACOs) receive shared savings by establishing a local healthcare delivery collaboration to coordinate across the full care continuum while improving care quality and reducing care costs below a specified benchmark.1 They are a promising approach to address care fragmentation and achieve the Triple Aim in the US health system.2-6
Increasingly widespread, ACOs cover about 10% of the US population.7 Congress established the Medicare Shared Savings Program (MSSP) as 1 of several ACO programs implemented by CMS to incorporate ACOs into Medicare.8 As of early 2018, there are 561 MSSP ACOs representing 10.5 million lives,9 making it the largest of Medicare’s ACO programs. CMS tracks MSSP ACO performance and reports that these ACOs have achieved reductions in cost while improving, and outperforming Medicare fee-for-service (FFS) providers on, most quality measures.10,11
Given the widespread nature of this delivery model, there is a need for evidence of its impact on healthcare quality. Longitudinal examinations of ACO quality have found that ACOs were associated with reduced utilization of postacute care (PAC), length of skilled nursing facility (SNF) stays,12 mortality,13 readmissions,14,15 and chronic obstructive pulmonary disease or asthma admissions16, and improved patient experience,17 chronic disease management, and preventive and pediatric care, especially among populationswith low socioeconomic status.18,19 Cross-sectional study findings further indicate that beneficiary race/ethnicity,20 ACO size and leadership style,21 provider mix,22 and urban/rural county23 are associated with quality.
Existing longitudinal literature focuses on comparing the ACO model with a counterfactual of other care models (namely, FFS), which is valuable and needed evidence given that the literature is still young and mixed. By comparison, given the expansion and prevalence of the MSSP, we examined key factors affecting within-ACO quality improvement in the program (ie, the counterfactual is an ACO compared with itself at an earlier time; we did not compare ACOs with non-ACOs), an area needing better evidence,21 so that we may understand how ACOs function maximally within the MSSP to improve quality, a key program goal.
Data Sources and Study Population
We utilized 2 data sources. First, we usedall publicly available MSSP ACO data files, containing quality scores, financial performance, and descriptive data on ACOs, their providers, and beneficiaries.24-27 Performance year 1 contains 220 ACOs from when they began (at certain points in 2012 or on January 1, 2013) through the end of 2013, and performance years 2, 3, and 4 contain 333, 392, and 432 ACOs, respectively, during calendar years 2014, 2015, and 2016. Second, we used proprietary data on ACO taxonomy type, geography, and risk-bearing arrangements from the Leavitt Partners ACO Database, which has tracked ACOs since 2010 using news releases, public reports, industry directories, surveys, and interviews.15,28-30 These were merged by MSSP ACO identifiers to construct a panel of ACOs from 2013 to 2016.
The objective was to examine how quality metrics changed within ACOs over time. This is represented by MSSP ACO quality performance metrics across 4 quality domains: patient/caregiver experience, care coordination/patient safety, clinical care for at-risk populations, and preventive health. Across 2013-2016, there were 44 measures (42 single measures plus 2 composites) that came from claims, patient survey, or ACO-reported data.31 In any given year, there were 33 to 34 single measures (some changed or were replaced over time). After careful review of measure definitions (see Table 1), in regression analyses we omitted certain years of measures before or after significant specification changes in a way that maximized data-years included but avoided spurious findings, a potential issue with the federal government’s early MSSP quality examinations.
We were primarily interested in 6 key independent variables related to ACO structure, function, relationships, and geography discussed or hypothesized in the literature as important to ACO care quality and success: 1) taxonomy (primarily led by a hospital, physicians, or both),32,33 2) percentile of risk-bearing maturity compared with other ACOs, 3) presence of a commercialcontract,34 4) expenditures on PAC,12,32,35,36 5) size (attributed beneficiaries), and 6) rurality.23 These were drawn or derived from the Leavitt Partners ACO Database (except size and PAC expenditures, which are publicly available).
Other variables can confound these relationships, so we included the following as covariates (ACO-level): beneficiaries’ age, sex, dual-eligible status, and Hierarchical Condition Category risk scores (to attempt case-mix adjustment); provider mix; patient-to-provider ratio; per capita benchmark; and market context (ratio of market cost relative to national cost).
First, we used linear regression to examine cross-sectional adjusted associations between average quality performance and covariates (between-ACO associations), and then we used fixed-effects linear regression to longitudinally examine quality improvement (withinACO associations). Fixed-effects methodology mechanically controls for time-invariant traits (measured or not).37 We regressed each quality measure onto all time-variant independent variables, covariates, and fixed-effects dummies for ACO and year. Only 2 key independent variables, PAC expenditures and size, had time-variant data. However, we incorporated key time-invariant traits by examining how significant PAC and size coefficients from fixed-effects models differ when stratified by ACO taxonomy, commercial contractpresence, rurality, and risk-bearing maturity (“subgroup analyses”). All analyses were conducted in Stata/SE 15.1 (StataCorp LP; College Station, Texas) using heteroskedasticity-robust standard errors.37
Table 2 presents descriptive statistics of quality metrics and independent variables by year and change scores (ie, differences in scores) between years. Almost all quality measures improved across the 4 years, except that roughly half of the patient/caregiver experience measures showed small decreases until year 4, when they increased. As many quality measures are assessed on different scales, our analysis also calculated relative change scores to mimic early CMS methods.38 We found similar but more conservative results than did early federal government reports,10,11 likely due to our study only focusing on trends involving no measure definition changes. The largest, most consistent quality improvements were in preventive health. Larger improvements were notable in heart failure admission rates, screening for fall risk, pneumonia vaccination, and screening/follow-up for clinical depression.
Table 3 shows key independent variable coefficients from linear regressions of each quality measure onto all covariates (comparison groups: ACO vs other ACOs). Table 4 shows statistically significant key time-variant independent variable coefficients (PAC expenditures change and ACO size change) from fixed-effects regressions of each quality measure with 2 or more years of data onto all time-variant coefficients and subgroup analyses, decomposing these coefficients by key time-invariant independent variables (comparison groups: ACO vs only itself at different times). Below, we summarize key stories from these tables.
First, cross-sectionally, there were significant positive associations between size and quality, particularly in clinical care for at-risk populations (Table 3). However, within-ACO change in size was inversely associated with quality changes, also notably among clinical care for at-risk populations (Table 4). Subgroup analyses showed that this inverse association varies substantially by ACO taxonomy and maturity, the highest magnitude being among physician-led ACOs and those less experienced in terms of program time and risk-bearing maturity. Figure 1 provides an illustrative example using 2013-2014 change scores in the coronary artery disease composite: Although most ACOs succeeded in improving quality, ACOs in the highest quartile of size change (+2289 to +40,091 beneficiaries) had much smaller improvements than those in the lowest quartile (–8209 to –294), varying substantially by ACO taxonomy and maturity.
Second, there is a clear relationship between PAC expenditures and quality. Cross-sectionally, there are consistent inverse associations between PAC expenditures and most quality measures, except for patient/caregiver experience measures (Table 3). Within-ACO changes in PAC expenditures had inverse associations with changes in care coordination/patient safety, notably all measures related to unplanned hospital admissions and readmissions (Table 4). Subgroup analyses showed this inverse association to generally be mitigated among physician-led ACOs and ACOs more experienced in terms of time in the program and risk-bearing maturity. Figure 2
provides an illustrative example using 2014-2015 change scores in all-cause readmissions: Although most ACOs decreased readmissions, ACOs in the highest quartile of PAC expenditure change (+0.5% to +10.5%) had much smaller decreases than those in the lowest quartile (–10.4% to –1.5%), varying substantially by ACO taxonomy and maturity.
The presence of commercial contracts or rural settings showed inconsistent findings in subgroup analyses, but we examined them and other time-invariant traits in cross-sectional results (Table 3). First, hospital-led ACOs had higher average quality in many measures of preventive care and clinical care for at-risk populations, although provider-led ACOs often had higher patient experience scores. Second, risk-bearing maturity had a consistent positive association with quality. Last, ACO rurality was generally associated with better quality, except in care coordination and patient safety.
This study examined in depth the first 4 years of MSSP data, finding 4 main quality improvement conclusions: 1) ACO quality appears to be broadly improving, highlighting the potential success of the MSSP model; 2) although ACO size was positively correlated with quality, ACOs experienced some quality challenges while growing; 3) PAC expenditures increased then decreased, which was associated with quality changes; and 4) these findings vary by key organizational traits. We discuss these findings below in the context of theory and prior work, then conclude with implications for practice, policy, and quality improvement in the broader healthcare system.
First, our findings generally mirror CMS’ broad conclusion that MSSP ACOs have improved most quality measures (with largest improvements in screening for falls risk, pneumonia vaccinations, and screening/follow-up for depression and blood pressure).10,11 Additionally, we found large relative improvement in preventable heart failure admissions. Why the largest gains were seen in these measures, notably preventive health, is not entirely clear, although others have found ACOs to outperform non-ACOs in preventive care disparities18,19 and the ACO model is designed to improve care coordination1 (many preventive care measures require coordinated screenings). Lastly, the very small decreases in patient experience (likely clinically insignificant) observed in the first 3 years may be due to high average scores (topped-off measures), reinforcing prior calls to develop cross-cutting patient-reported outcomes that better reflect patient experience. Regardless, others have found that ACOs outperform non-ACOs in patient experience measures that are thought to be within a provider’s ability to control,17 and we found patient experience measures to improve in year 4.
Second, MSSP ACOs experienced large growth in beneficiaries in the first 3 years of the program (plateauing in the fourth), which was associated with quality. Larger size in any given year was associated with higher quality in more than one-fifth of measures (nonsignificant measures trended similarly), most in clinical care for at-risk populations, likely via larger economies of scale and more well-developed infrastructure and referral networks to handle complex patients. The process of ACO growth, however, was negatively associated with clinical care for at-risk populations. This could be explained by multiple startup cost mechanisms, including increased demand for resources to engage, attribute, and manage new beneficiaries; increased beneficiary-to-provider ratio; and any provider consolidation that growth may bring. The magnitude of this effect was generally largest among ACOs led by physicians and those with less program experience and risk bearing, potentially due to smaller average size (fewer economies of scale and less ability to absorb startup costs).
Third, the average ACO experienced increased initial PAC expenditures, presenting quality improvement challenges, but was able to later reduce these expenditures. Changes in PAC expenditures in any given year were associated with inverse changes in quality for two-thirds of measures. Within-ACO changes in PAC spending tended to affect preventable admissions and all-cause readmissions. ACO affiliation has been linked to reduced PAC utilization relative to non-ACOs,12 but nonetheless, PAC is a large source of cost variation, and close coordinated care and referral partnerships with SNFs and other PAC facilities have been flagged as a potential locus for ACO quality improvement and cost reduction.12,32,35,36 From subgroup analyses, we suggest that this opportunity may be the biggest for hospital-led ACOs.
Last, although limited to cross-sectional analyses, we gleaned insight into how ACO taxonomy, risk-bearing maturity, and rurality are associated with average quality. First, hospital-led ACOs performed better on measures of preventive health and clinical care for at-risk populations, perhaps because many require on-site screening and specialists to which smaller provider-led ACOs may not have direct access or because hospital-led ACOs may have more formal quality control programs reminding providers of preventive health requirements. We found that provider-led ACOs often had higher patient/caregiver experience scores. Although the reasons for this are unclear, provider-led ACOs had the smallest median size, and smaller practices are theorized to create a more personal setting that patients may prefer.39 Second, higher quality was consistently associated with ACOs that were more mature in terms of total contracts, program time, and risk bearing. Although this finding is likely partly influenced by survivorship of more advanced or experienced ACOs, it also could indicate the impact that having more advanced contract arrangements has on quality. ACOs with 1 or more commercial contracts are documented as having higher quality than those with only public contracts, which may be due to greater use of disease-monitoring tools, patient satisfaction data, and quality improvement activities, and larger provider compensation mechanisms for improved quality,34 although we observed that commercial contracts alone did not generally affect quality. Last, rural ACOs often had higher quality, which early evidence supports,23 although underlying mechanisms are unclear. This finding is promising, considering the unique challenges faced by rural ACOs, including a smaller number of covered lives spread out over large geographies (which limits ability to absorb cost variation) and infrastructure and data analytic capabilities less equipped to effectively implement population health management.23 Additional research is needed.
These findings must be interpreted within this study’s limitations. First, our analyses did not compare ACOs with non-ACOs but instead compared ACOs with either other ACOs or with themselves at different time periods within the MSSP. Thus, our findings cannot make assertions about how quality functions in the ACO model versus other models, such as FFS. Instead, given the increasingly widespread prevalence of accountable care models over the last several years, our goal was to examine how quality changed for organizations within the MSSP over time in order to understand how to capitalize on successes and avoid challenges moving forward.
Second, although incomplete quality reporting occurs in ACOs, the MSSP tracks and strongly disincentivizes incomplete quality reporting by withholding shared savings for ACOs that do not meet the quality reporting standard. We examined this in a sensitivity analysis and found that it was very uncommon for ACOs to fail to meet this standard and that failure became increasingly rare in each subsequent program year, so we suspect minimal bias here.
Third, bias could be introduced by virtue of which ACOs are likely to enter and exit the program, although we performed a sensitivity survival analysis and found no association between quality scores and program survival.
Fourth, although we cannot definitively assert the causal direction of our main findings, changes in our key time-variant variables (ACO size and PAC expenditures) more likely precede changes in quality and not vice versa. Regarding growth, we examined data over the first 4 years of the MSSP when ACOs were ramping up (plateauing in year 4). PAC expenditure presents possible endogeneity given that some quality measures are related to PAC (eg, readmissions), although many have argued that PAC precedes costs and patient outcomes, including recent work using instrumental variable analysis to circumvent selection bias and directionality issues.40 Nonetheless, further research is needed into how and why ACOs are growing and changing PAC expenditures and how these changes subsequently affect quality. Finally, although time-invariant confounding is ruled out in fixed-effects models, and in all models we have taken care to include both time-variant and time-invariant controls known or hypothesized to affect quality, we cannot definitively rule out confounding.
Most MSSP ACOs improved most quality measures over the first 4 years of the MSSP. The MSSP is a prevalent and expanding program that is, on average, achieving its quality improvement goals. Further, our companion study found that MSSP ACOs are achieving savings by shifting expenditures away from costly inpatient and long-term services to primary care provided in physician offices.41 That MSSP ACOs serve the primarily older and more chronically ill Medicare population indicates a success, suggesting that the MSSP model would likely be successful if expanded to other settings (public and private). Given this quality improvement success, the MSSP could more strongly incentivize more advanced risk‑sharing (beyond 1-sided, moving to MSSP “Track 3”). Moreover, the finding that most ACOs improved quality shows how much room for improvement there is: Although the shared savings model offers major improvements over traditional/previous FFS models, it is still an alternative payment model (APM) built and dependent on FFS architecture. Continuing to move down the advanced APM path toward population-based payments (ie, care is prospectively contained within a single payment over a fixed time, linked to quality, value, and patient-centeredness) may further maximize value-
There exist clear opportunities for ACO quality to further improve. First, ACOs experienced tremendous growth in the first 4 years of the MSSP, and although larger size was associated with higher average quality, growth presented minor quality challenges in caring for at-risk patients; in other words, growing cautiously as resources permit, not aggressively, may be a good strategy for ACOs. Further, concerns over economies of scale and administrative burden associated with becoming or expanding an ACO could accelerate provider consolidation, previously raised as an unintended consequence of recent health reform. Consolidation can increase shared resources and care coordination, but it has been associated with increased prices without improved quality due to reduced market competitiveness.43-45 The number of ACOs continues to grow, and ACO size is just beginning to plateau, so continued public and private funding for ACO infrastructure development will be beneficial to maintaining quality gains and minimizing potential negative consequences, especially for ACOs serving rural or underserved populations (eg, the ACO Investment Model46). Second, developing stronger, more coordinated partnerships with SNFs and other PAC facilities (bringing them more formally into ACO networks) likely would improve care quality and cost. Finally, ACOs could benefit from capitalizing on their diversity (quality challenges and strengths varied by ACO leadership type, risk-bearing ability, and rurality), presenting shared learning opportunities.
Taking advantage of these opportunities could maximize ACO model expansion and quality improvement nationwide. More work still needs to be done, however, to better understand how and why the above trends, findings, and implications play out in ACOs and to share learnings with the broader system, as ACOs, advanced APMs, and the health system as a whole evolve.
The authors thank Lia Winfield, PhD, of Leavitt Partners, for her assistance in reviewing the submitted manuscript.
Dr Pham’s work on this manuscript was done while she was a Senior Policy Fellow at Duke University’s Robert J. Margolis, MD, Center for Health Policy (Washington, DC).