Jennifer Peterson, MHA

 The evolution in the accountable care organization (ACO) payment model initiatives by CMS reflects an increased understanding of the benefits and hurdles of using such an advanced payment model. As the Medicare Access and CHIP Reauthorization Act requires Medicare providers to choose an incentive-based payment track, which can include participation in an ACO, understanding what is achievable in the different types of ACO models is important. The most recent version, the Next Generation ACO (NGACO) model, increases both the risk and financial reward that providing organizations can realize relative to other types of Medicare Shared Savings Program (MSSP) ACOs. This study aims to examine the differences in savings and quality scores across MSSP ACOs and NGACOs.

 Observational retrospective study of ACO-level data.

In this study, we used panel data to estimate fixed effects regressions comparing ACO savings and quality scores across the 2 ACO types (NGACO and MSSP) using publicly accessible aggregated ACO data. We studied 737 unique ACOs (680 MSSP ACOs and 74 NGACOs in our sample, with 17 ACOs switching from one type of ACO to the other) from 2016 to 2018.

 On average, the NGACOs had more aligned beneficiaries, but no statistically significant differences emerged in average gross savings ($1.90 million for NGACOs vs $2.21 million for MSSP ACOs; 

 = .78) after adjusting for size and fixed effects. We also found mostly insignificant differences across 37 quality measures used to calculate the share of savings that ACOs receive.

 NGACO and MSSP cost and quality data show similar performance, suggesting that increasing financial risk to health systems may not affect performance.

The Affordable Care Act in 2010 established an innovation center within CMS and facilitated the testing of various payment reform models focused on improving patient care and population health without increasing costs and lowering them where possible.

 As part of these efforts, CMS rolled out the Medicare Shared Savings Program (MSSP) in 2012, which allowed eligible providers to create accountable care organizations (ACOs) with the hope that increased coordinated care would result in savings that could be shared as long as quality standards were met.

As evidence emerged about how ACOs worked and their effectiveness compared with prior payment models in both cost and quality, CMS has tested and refined additional aspects of the ACO programs.

 The Pioneer ACO model included providers that largely already had the requisite infrastructure and care coordination practices in place, and they were able to show improvements in quality, but evidence on cost savings was mixed.

 CMS also initiated the broader MSSP in 2012, with less stringent risk-sharing requirements. To address concerns about the significant start-up costs for an ACO without that existing infrastructure, CMS designed a Medicare Advance Payment (AP) ACO model that provided an up-front fixed payment to participating ACOs and then a monthly payment thereafter based on the number of attributed beneficiaries. These advanced payments were to be recouped from shared savings in the first agreement period only. The Medicare ACO Investment Model (AIM) then built on the AP ACO model by allowing more ACOs (entering MSSP in 2012-2016) to defray the capital investments necessary to form and maintain an ACO. Empirical evidence to date on these advanced alternative payment models (APMs) is mixed, with the AP model demonstrating little effect on quality and an overall increase in total spending per beneficiary per month (PBPM),

 whereas the AIM has shown savings of $381.5 million, but the program includes only 14 (of 47) AIM ACOs still in the shared savings program.

 Nonetheless, since the initiation of the MSSP, ACO participation has grown to 517 ACOs with 11.2 million aligned beneficiaries nationwide in 2020.

The Next Generation ACO (NGACO) model, rolled out in 2016, offered higher levels of risk and reward than earlier ACO models, mechanisms that enabled a graduation from traditional fee-for-service reimbursements to population-based payments, and tools to support better patient engagement and care management. One important goal of the NGACO model, as stated by CMS, “is to test whether strong financial incentives for ACOs, coupled with tools to support better patient engagement and care management, can improve health outcomes and lower expenditures for Original Medicare fee-for-service beneficiaries.”

 This model, which will run through 2021, has more than 45,000 providers and more than 1.2 million aligned beneficiaries.

 The evaluation results to date suggest divergent effects in the first 2 years, with nearly $117 million in increased spending among the 2016 cohort of participating ACOs and more than $86 million in reduced spending among the 2017 cohort.

 Further understanding the relationships between increased financial risk and performance in cost and quality between MSSP and NGACO programs may have important implications for health systems and individual health care providers in terms of deciding to accept varying levels of risk in these advanced APMs.

In this study, we examined NGACOs using publicly available ACO data from CMS that included NGACOs and MSSP ACOs from 2016 to 2018. We aimed to investigate the extent to which ACO performance on reported cost and quality measures varied for beneficiaries participating in NGACOs relative to those participating in the lower-risk MSSP ACO models.

We used publicly available data from 2016 to 2018 from CMS that aggregated all quality measures, net costs, and savings rates from all MSSP ACOs. We included NGACOs and all other MSSP ACOs with active agreements, hereafter denoted as an ACO type of “NGACO” and “MSSP,” respectively.

We focused on outcome measures that are reported consistently across the different ACO types, which included 37 quality measures categorized by CMS under patient/caregiver experience, care coordination/patient safety, preventive health, and at-risk population. We also examined a key cost measure: gross savings/losses, calculated as the total benchmark expenditures minus the total aligned beneficiary expenditures. Benchmark expenditures in the NGACO model are set each performance year prospectively, using expenditures, risk scores, and quality measures from a 1-year historic baseline that is regionally detrended with the intention to avoid penalizing successful performance.

 For the other MSSP ACOs, benchmarks can vary based on agreement start date and risk track (1- or 2-sided) but broadly weigh more heavily on an ACO’s recent expenditures without the regional detrending.

 For either type of ACO, if expenditures on all ACO-aligned beneficiaries exceed that benchmark, it will have losses. We included the number of aligned beneficiaries and the calendar year as covariates in regression models.

We used panel data analyses, accounting for ACOs that we observed over multiple years. We calculated descriptive statistics and compared means using 2-sided 

We also estimated linear regressions to adjust for the number of aligned beneficiaries in each ACO and the calendar year. Finally, we estimated fixed effects regressions to adjust for time-invariant ACO characteristics—for example, the state or health care market in which the ACO is operating or the ACO’s key partners (to the extent that these characteristics did not change). The regressions followed this form:

 is an indicator for ACOs participating in the NGACO program, 

 is a continuous measure of the number of attributed beneficiaries, 

 is a vector of ACO fixed effects. To adjust for multiple comparisons, we applied a Bonferroni correction and considered differences statistically significant with a 

 value < .002. All analyses were performed using Stata version 16 (StataCorp).

In our sample, there were 737 unique ACOs over the 3-year period; 17 switched from one type of ACO to the other. Thus, there were 680 MSSP ACOs and 74 NGACOs in our sample (see 

). On average, NGACOs had about 9000 more aligned beneficiaries than MSSP ACOs (

 < .001). The savings rate was between 1% and 2%, on average, and not statistically different across the 2 groups, even after adjusting for differences in the number of aligned beneficiaries and calendar year and in our fixed effects estimation. NGACOs appeared to have higher average gross savings, at about $4.5 million, relative to MSSP ACOs at $2.38 million (

 = .04), but once we adjusted for the ACO size and calendar year and ACO fixed effects, this difference became smaller in magnitude and statistically insignificant (adjusting for ACO size/year: 

 = .82; adjusting for ACO size/year and ACO fixed effects: 

, we present adjusted score components in the patient/caregiver experience domain based on the fixed effects regression results. Scores are not significantly different by ACO type, except that the Consumer Assessment of Healthcare Providers and Systems patients’ rating of providers is slightly higher among NGACOs relative to MSSP ACOs, but after adjusting for multiple comparisons, this does not meet our level of significance (

Next, we examined adjusted score components in the care coordination and patient safety domain. In 

, we present the predicted means from the fixed effects regression results. Only the difference in ACO8, the risk-standardized measure for all-condition readmissions, approaches statistical significance (

 = .019), with NGACOs having slightly higher readmission rates.

, we present adjusted score components for preventive health and at-risk populations. Here we found marginally significant differences for some at-risk population measures. NGACOs had a lower percentage of beneficiaries with depression with remission at 12 months

 = .016) and lower use of aspirin or another antiplatelet among patients with ischemic vascular disease (

Our analyses suggest no significant difference in gross savings/losses between MSSP ACOs and NGACOs after adjusting for the number of aligned beneficiaries and ACO fixed effects. Also, no consistent differences emerged between the 2 payment models in most quality measure domains.

Our findings have significant potential implications for both health care systems and health care providers participating in single-sided vs double-sided risk advanced APMs. The lack of difference in gross savings/losses and quality measure domains between MSSP ACOs and NGACOs may influence the decision of an ACO’s leadership to participate in either of these programs. Our analysis suggests that, on average, gross savings/losses between MSSP and NGACO models were not significantly different and that other differences between programs (such as risk to an ACO) may be more important factors in determining program participation. In contrast, it is unclear if individual providers may be influenced to join or remain in a payment model with an increased risk like an NGACO, as individual providers’ exposure to such risk/rewards are likely influenced by other factors not evaluated in this study (eg, employed vs affiliated provider contracts, gainsharing).

Our work differs from most of the CMS-sponsored evaluation studies of ACOs because we are not using individual beneficiary-level data, but our findings are consistent with several of the studies examining earlier CMS ACO models.

 A final evaluation of the NGACO model is still under way, but our findings suggest very few differences in the key outcomes that will be used for generating savings payments between the NGACO and MSSP models. However, we note that selection is still a significant concern, as the providers and organizations that have chosen to participate in an MSSP ACO or NGACO are likely very different from those who did not but who may be considering forming one in response to the Medicare Access and CHIP Reauthorization Act (MACRA). As earlier research has found, there are significant start-up and infrastructure costs and hurdles to forming an ACO, which may influence which providers decide to participate in an ACO vs choosing the Merit-based Incentive Payment System payment track to receive reimbursement for their Medicare beneficiaries.

Our analysis includes the following limitations: First, we do not observe data for all ACOs at all points in time and we have no pre-ACO data, which would allow us to explore and possibly adjust for potential selection effects. Second, although we are able to estimate fixed effects models to adjust for time-invariant characteristics of the ACOs, ACOs likely adapted their behaviors over time in ways that would affect their performance and quality measures. Thus, our work should be considered exploratory and descriptive and not causal. Nonetheless, understanding to what extent there might be differences is useful for providers considering how to respond to MACRA. We also note that some of CMS’ quality measures are not risk adjusted (eg, patient experience and process measures).

The results of this study provide evidence that ACOs in the NGACO model perform similarly to organizations participating in the MSSP model on publicly reported quality and cost measures. These findings suggest that ACO models with increased financial risk to providers may not significantly affect performance on such cost and quality measures.


 Department of Internal Medicine (JDC, KAK, JF) and Department of Population and Data Sciences (KAK), University of Texas Southwestern School of Medicine, Dallas, TX; Southwestern Health Resources (JDC, JL, RK, JP, JF), Dallas, TX; RAND Corporation (KAK), Arlington, VA.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (JDC, JL, RK, JP, JF); acquisition of data (JDC, RK, JP, JF); analysis and interpretation of data (JDC, KAK, RK, JP, JF); drafting of the manuscript (JDC, KAK, JL, RK, JP, JF); critical revision of the manuscript for important intellectual content (JDC, KAK, JL, JF); statistical analysis (KAK, RK, JP); administrative, technical, or logistic support (JDC, JL, JF); and supervision (JDC, JF).

 John D. Clark, MD, PhD, University of Texas Southwestern School of Medicine, 8150 Brookriver Dr, 5th Floor, Dallas, TX 75247. Email: JohnD.Clark@UTSouthwestern.edu.

1. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. 

. 2008;27(3):759-769. doi:10.1377/hlthaff.27.3.759

2. Medicare Learning Network. Summary of final rule provisions for accountable care organizations under the Medicare Shared Savings Program. April 2014. Accessed November 4, 2020. https://www.kff.org/wp-content/uploads/sites/2/2014/04/aco_summary_factsheet_icn907404.pdf

3. MACRA. CMS. Updated November 18, 2019. Accessed August 25, 2020. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/MACRA-MIPS-and-APMs.html

4. Colla CH, Lewis VA, Kao LS, O’Malley AJ, Chang CH, Fisher ES. Association between Medicare accountable care organization implementation and spending among clinically vulnerable beneficiaries. 

. 2016;176(8):1167-1175.
doi:10.1001/jamainternmed.2016.2827

5. Pham HH, Cohen M, Conway PH. The Pioneer accountable care organization model: improving quality and lowering costs. 

. 2014;312(16):1635-1636. doi:10.1001/jama.2014.13109

6. McWilliams JM, Hatfield LA, Chernew ME, Landon BE, Schwartz AL. Early performance of accountable care organizations in Medicare. 

. 2016;374(24):2357-2366. doi:10.1056/NEJMsa1600142

7. Nyweide DJ, Lee W, Cuerdon TT, et al. Association of Pioneer Accountable Care Organizations vs traditional Medicare fee for service with spending, utilization, and patient experience. 

. 2015;313(21):2152-2161. doi:10.1001/jama.2015.4930

8. Tu T, Muhlestein D, Kocot SL, White R. The impact of accountable care: origins and future of accountable care organizations. Brookings. May 2015. Accessed August 25, 2020. https://www.brookings.edu/wp-content/uploads/2016/06/Impact-of-Accountable-CareOrigins-052015.pdf

9. L&M Policy Research. Evaluation of CMMI accountable care organization initiatives: advance payment ACO final report. CMS. November 25, 2016. Accessed August 25, 2020. https://innovation.cms.gov/files/reports/advpayaco-fnevalrpt.pdf

10. Abt Associates. Evaluation of the Accountable Care Organization Investment Model. CMS. September 2020. Accessed September 2020. https://innovation.cms.gov/data-and-reports/2020/aim-final-annrpt

11. Shared Savings Program fast facts—as of January 1, 2020. CMS. 2020. Accessed August 25, 2020. https://www.cms.gov/files/document/2020-shared-savings-program-fast-facts.pdf

12. Next Generation ACO Model. CMS. Updated October 2, 2020. Accessed August 25, 2020. https://innovation.cms.gov/initiatives/next-generation-aco-model/

13. Next Generation ACO Model. CMS. Updated October 2, 2020. Accessed August 25, 2020. https://innovation.cms.gov/innovation-models/next-generation-aco-model

14. Next Generation ACO Model benchmarking methods. CMS. December 15, 2015. Accessed August 25, 2020. https://innovation.cms.gov/files/x/nextgenaco-methodology.pdf

15. Liu L, Svoboda R, Zhang Y. Using a modified Next Generation ACO benchmark can improve the MSSP. 

16. Rose S, Zaslavsky AM, McWilliams JM. Variation in accountable care organization spending and sensitivity to risk adjustment: implications for benchmarking. 

. 2016;35(3):440-448. doi:10.1377/hlthaff.2015.1026

17. MH-1 (NQF 0710): Depression Remission at Twelve Months. CMS. 2018. Accessed August 25, 2020. https://qpp.cms.gov/docs/QPP_quality_measure_specifications/Web-Interface-Measures/2018_WI_MH1.pdf

18. IVD-2 (NQF 0068): Ischemic Vascular Disease (IVD): Use of Aspirin or Another Antiplatelet. CMS. 2018. Accessed August 25, 2020. https://qpp.cms.gov/docs/QPP_quality_measure_specifications/Web-Interface-Measures/2018_WI_IVD2.pdf

Articles

2016-2018 Next Generation Accountable Care Organization (ACO) and Medicare Shared Savings Program cost and quality data show similar performance, suggesting that increasing financial risk to health systems may not affect performance.

Jennifer Peterson, MHA

Articles

Differences in Savings and Quality by Type of ACO Model