ACOs With Risk-Bearing Experience Are Likely Taking Steps to Reduce Low-Value Medical Services

Margje H. Haverkamp, MD, PhD; David Peiris, MD, PhD; Alexander J. Mainor, JD, MPH; Gert P. Westert, PhD; Meredith B. Rosenthal, PhD; Thomas D. Sequist, MD, MPH; and Carrie H. Colla, PhD

Accountable care organizations (ACOs) are voluntary groups of provider organizations that are collectively held accountable for both quality of care and total spending for a defined group of patients through payment contracts. A promising strategy to improve quality and financial sustainability involves the reduction of low-value medical services. Indeed, prior research shows a modest decrease in the use of low-value care, and thus in spending rates, for ACOs compared with non-ACO providers with predominantly fee-for-service payment models.1 While utilization and related spending have decreased in ACOs, quality scores and care satisfaction have remained similar or improved compared with other organizations.2-4 However, it is not clear what strategies ACOs deploy to lower unnecessary care, nor what features predict a commitment toward overuse reduction.

One way to tackle low-value care is to embrace the Choosing Wisely campaign.5 Choosing Wisely aims to reduce the delivery of low-value medical services by promoting conversations between patients and physicians on the appropriateness of care. More than 70 US specialty societies have defined concise lists of 5 to 10 wasteful interventions that “physicians and patients should question.”5 The synergies of ACOs and Choosing Wisely regarding care improvement and overuse reduction suggest that ACOs committed to reducing low-value care should be aware of this campaign and also work toward actively lowering the utilization of these medical services. In this study, we analyze data from the National Survey of ACOs (NSACO) to determine which strategies are used to reduce low-value care and identify the ACO characteristics that predict the use of such methods.



The NSACO is an online survey designed by researchers at the Dartmouth Institute for Health Policy and the University of California, Berkeley. It questions ACOs (Medicare Shared Savings Program [MSSP] ACOs, Medicare Pioneer ACOs, state Medicaid ACOs, and commercial payer ACOs) on their composition, characteristics, contracts, and capabilities.6,7 A total of 752 ACOs were identified through public documents, provider surveys, scientific literature, and certification by the National Committee for Quality Assurance and invited to participate. Senior ACO executives, including chief executive officers, executive directors, and chief medical officers, filled out the survey. At the time of our analysis, the survey was fielded in 3 consecutive waves, with each wave questioning newly formed ACOs (wave 1, October 2012-May 2013; wave 2, September 2013-March 2014; wave 3, November 2014-May 2015). The median duration between the implementation of the ACO contract to the time of the survey was 11.6 months (interquartile range, 7.1-13.2 months).7 Over all 3 waves, 64% of ACOs filled out the survey. Waves 2 and 3 of the survey were more elaborate, and therefore, ACOs from wave 1 were approached with a follow-up survey asking additional questions during the fielding of wave 3. Questions about Choosing Wisely were not asked in wave 1, excluding from our analysis 93 ACOs that participated in the first wave but not the follow-up survey. Survey questions related to Choosing Wisely included: (1) “Are you aware of the Choosing Wisely program?” and, if the response was positive, (2) “What steps have you taken to reduce the use of Choosing Wisely tests and procedures?”

Multivariate Statistical Modeling

We divided our sample into 2 groups: (1) ACOs not aware of the Choosing Wisely campaign or aware but not taking steps to support it and (2) ACOs taking steps to actively reduce the use of low-value medical services.

We compiled an a priori list of 62 survey responses that could be associated with the decision to take steps to support Choosing Wisely (the eAppendix Table [eAppendix available at] lists these characteristics). Based on existing hypotheses about how these characteristics might affect an ACO’s decision to take steps to reduce overuse, as well as on simple pairwise significance tests, we then selected a subset of 22 variables from this list. We excluded ACO characteristics on quality behavior to prevent potential reverse causality with waste-reducing efforts. To identify the main drivers behind the decision to take steps to reduce overuse within those 22 variables, we used both logistic regression (stepwise regression and LASSO regression) and classification techniques (random forest).8 Stepwise logistic regression was performed both backward and forward. LASSO imposes shrinkage constraints on the variables, resulting in an optimal model with only those characteristics that have a coefficient greater than 0. Random forest stratifies the predictor space in regions with nonlinear boundaries between variables, producing multiple decision trees that are combined into a single consensus prediction. The 3 statistical approaches identified 3 sets of prediction variables, and we subsequently assessed consistency of associations across these 3 models. Furthermore, we evaluated the relative predictive merits of each model by comparing their receiver operating characteristic (ROC) curves and confusion matrices on the basis of their implied misclassification rates (fraction of false positives and false negatives).

Savings and MSSP Quality Scores

CMS publicly reports shared savings payments and the outcomes of 33 quality measures per performance year for each participating ACO. The quality scores are in 4 domains: patient experience (including a measure on shared decision making), care coordination, at-risk measures, and preventive care. We calculated an overall quality score and a quality score per domain for the first 2 performance years of each ACO (the year in which the ACO filled out the survey) using the CMS sliding scale approach, as described elsewhere.7,9 We compared these quality scores and the savings per beneficiary attributed to the ACO according to CMS in the first 2 years with reference to historical expenditure benchmarks, between ACOs taking steps to reduce overuse and ACOs not taking steps, using a 2-sample t test.

Characteristics of ACOs Taking Steps to Reduce Overuse

Of 305 potential ACOs (wave 1 follow-up, 82 ACOs; wave 2, 95 ACOs; wave 3, 128 ACOs), survey respondents for 267 ACOs answered the question “Are you aware of the Choosing Wisely program?” Of these, 58% (155) reported awareness of Choosing Wisely, but only 31.5% (84) said they had also taken steps to reduce the use of low-value services (the eAppendix Table characterizes these ACOs). Consequently, 183 ACOs (68.5%) did not take such steps, partly because they were not aware of the Choosing Wisely campaign.

Compared with ACOs not taking steps, ACOs that had implemented strategies to reduce waste included more hospitals in their largest contract (P <.01) and were significantly more likely to consider themselves an integrated delivery system (P <.01) (the eAppendix Table compares these 2 groups). Provider organizations within ACOs implementing those strategies had more often jointly pursued risk-based payment contracts in the past (P <.01), and this group of ACOs had previously participated in a higher number of payment reform efforts than ACOs that did not take steps (P <.01). More ACOs taking steps to reduce waste had at least 1 commercial contract (<.01), and Medicare contracts were less prevalent (P = .03). Although their commercial contracts were more often characterized by both bonus and downside risk (P = .02) than by a bonus only (= .03), this bonus was less often contingent on quality metrics (P = .02). In addition, ACOs that actively reduced low-value services were more likely to allocate shared savings bonus payments across participating members (P = .04), compensate physicians based on clinical quality measures (P = .01), and share cost measures among their physicians (P = .02). A larger proportion of ACOs that were not aware of Choosing Wisely or not actively reducing waste were safety-net organizations (P <.01), defined as having more than 25% uninsured or Medicaid beneficiaries.

Of the 84 ACOs that reported using strategies to reduce low-value care, just 57 (68%) were asked to specify the steps they had taken, because this question was added to the survey after wave 2. On average, these ACOs took 3 different steps. The most frequently reported waste-lowering strategies included educating physicians on low-value tests and procedures (82.5% of ACOs), encouraging discussions between physicians and patients about appropriate care (73.7%), and disseminating Choosing Wisely materials (68.4%) (Table 1). Other strategies included handing out decision guides for patients, pop-ups in the electronic health record reminding physicians of the low value of certain tests and procedures, and audit and feedback on individual physician performance. Few ACOs (3.5%) reported taking steps to reduce waste by changing physician payment incentives.

Predictors for Implementing Strategies to Reduce Low-Value Care

Backward and forward stepwise logistic regression identified the same 6 variables and similar importance for predicting the use of low-value care–reducing strategies (Table 2). The presence of at least 1 commercial contract and prior participation of all provider organizations within the ACO in risk-based payment contracts were the most important factors. These were also the 2 most significant predictors in the LASSO model (eAppendix Figure 1). This model gave only 1 negative predictor, which predicted that an ACO would not actively seek waste reduction. The characteristic, more than 25% uninsured or Medicaid patients, was fourth in order of importance, and also 1 of the 6 characteristics in the stepwise regression (Table 2). In the consensus prediction tree from the random forest, the third model we used, the same 2 variables (at least 1 commercial contract, prior joint experience with risk-based payment contracts) were the 2 highest branches in the decision tree (Figure). Similarly, having more than 25% uninsured or Medicaid patients was the most influential negative predictor in this model. In conclusion, our working model to predict which ACOs would deploy strategies to reduce low-value care consisted of 2 positive predictors indicating joint experience in risk taking in the form of financial models, as well as 1 negative predictor (namely, a large contingent of safety-net patients).

The 3 models exhibited similar minimum misclassification rates on their ROC curves. The lowest possible misclassification rate for both stepwise regression and LASSO was 24% (eAppendix Figure 2 shows the true- and false-positive rates for the minimum misclassification of each model), whereas it was 26% for random forest (data not shown). This attributes to these models a power to identify 33% to 39% of true positives, or ACOs that take steps to reduce low-value care.

Relationship Between Taking Steps and CMS Data on Quality and Savings

Waste-reducing strategies were not associated with differences in CMS quality measures or savings per beneficiary for the 158 ACOs for which CMS data were available in performance year 1 (46 taking steps, 112 not taking steps) (Table 39). Furthermore, changes in these parameters from performance year 1 to year 2 were similar (32 and 70 ACOs, respectively; data not shown).

Based on 3 waves of the NSACO, we show that just one-third of all ACOs are taking steps to reduce low-value care. The best predictors for an ACO to deploy strategies for waste reduction were consistent across the 3 models with various statistical approaches. ACOs with a commercial contract and those whose provider organizations have jointly pursued risk-based payment contracts in the past are more likely to actively reduce overuse. Our finding is consistent with those of a recent study wherein ACOs with commercial payer contracts were more actively implementing efficiency measures compared with ACOs with public payer contracts only.7

We found that ACOs with a large contingent of uninsured or Medicaid patients probably do not take steps to reduce overuse. However, the delivery of low-value care is as common among uninsured or Medicaid patients as among the privately insured.10 In prior research, minorities and those with poor and fair health were notably at a higher risk of receiving wasteful medical services.11,12 Both groups are overrepresented in the Medicaid population.13,14 Therefore, safety-net ACOs should pay attention to overuse of medical services. However, with relatively fewer resources for quality improvement, these ACOs may be prioritizing tackling the underuse of high-value practices over limiting low-value practices.

Furthermore, we did not find correlations between taking steps to reduce overuse and CMS quality measures (including the use of shared decision making) or between such steps and overall savings for ACOs participating in the MSSP in the first 2 performance years. Indeed, although waste in healthcare is ubiquitous,15 quality measures and savings are determined by many factors other than waste reduction alone. In addition, the effect of strategies other than clinical decision support and performance feedback to lower low-value care may be limited, certainly if they do not address both patient and provider roles.16 For example, audit-and-feedback methods are efficient in reducing unnecessary antibiotic prescriptions, but only if the intervention is perpetuated.17 Decision aids—but not shared decision making or the dissemination of educational materials—have moderate effect on the use of discretionary surgery.18 Although not yet extensively studied, certain pay-for-performance models have resulted in only modest improvements in care processes and outcomes.19 In our study, just half of ACOs aware of Choosing Wisely took active steps to reduce low-value care, and in most of those ACOs, the interventions were low impact and did not include the most promising strategies for waste reduction (namely, decision aids, computer decision support, and audit and feedback on individual physician behavior). To be meaningful, increasing awareness of Choosing Wisely should go hand in hand with practical advice for provider organizations on how to enhance appropriate use of care efficiently. Therefore, more research is needed to determine the correct design of strategies to reduce low-value care, the potential to lower utilization of such care in ACOs, and the numeric effect of reductions on quality and savings in the long run.


Several limitations to studies involving NSACO data have been recognized,6,7 notably, the fact that survey questions were answered by ACO executives who may not have been aware of efforts to reduce low-value care in their organization and the short period (less than 1 year) between the start of the ACO contract and the survey. Within this time frame, ACOs may not yet have been able to initiate strategies to reduce low-value care. Of note, previous comparisons of ACOs filling out the NSACO survey with those that failed to respond have not shown a significant nonresponse bias in terms of beneficiary or provider composition, organizational structure, overall quality, or savings distribution.6,20,21

Physicians are an important source of low-value care utilization. It explains the focus chosen by Choosing Wisely on conversations between physicians and patients on appropriate care. However, collective risk taking in financial contracts influences ACOs to actively reduce low-value care more than physician leadership does. It is also possible that other unobserved characteristics in linkage with risk-bearing experience are influencing an ACO’s decision to seek a reduction in low-value care. In addition, reverse causality (ACOs most confident in their capacity to reduce waste signing risk-bearing contracts) cannot be excluded. However, this would not negatively affect the relationship we detected between collective risk taking and waste reduction.


The positive influence of risk bearing on efforts to reduce low-value care may be explained by a combination of resources, incentives, and opportunity. First, ACOs contracting with commercial payers may have more stringent contracting requirements that force them to prioritize waste reduction and more resources dedicated to quality improvement than do ACOs involved exclusively in public payer contracts. Second, ACOs with prior experience in risk-based contracting may have been able to develop the culture, systems, and technical know-how to tackle challenging issues such as overuse over time. Consequently, ACOs with less experience in risk bearing and safety-net ACOs will likely start prioritizing the reduction of overuse as they acquire more risk. In the meantime, those ACOs should be otherwise stimulated to reduce overuse and specifically targeted by advocacy efforts of healthcare payers and the Choosing Wisely campaign. Furthermore, researchers, policy makers, and the Choosing Wisely campaign should focus on defining waste-reducing efforts that are efficient and practical in use, with special attention to audit-and-feedback mechanisms on individual physician performance for both underuse and overuse. With US healthcare spending as high as 18% of gross domestic product, and healthcare outcomes in the United States lagging behind in comparisons with other high-income countries,22 all ACOs should use available levers for waste reduction, including Choosing Wisely materials, and implement strategies with proven efficacy in reducing low-value care to decrease costs and increase the quality of healthcare.


Dr Haverkamp and Dr Peiris, both Harkness fellows 2015-2016, gratefully acknowledge their sponsor, the Commonwealth Fund (New York), which has also supported the work with grant 20160616. In addition, Dr Haverkamp recognizes the Dutch Ministry of Health Welfare and Sport for sponsoring her fellowship. Furthermore, this project is supported by Agency for Healthcare Research and Quality (AHRQ) grants R01HS023812 and U19HS024075. The latter grant falls under AHRQ’s Comparative Health System Performance Initiative, which studies how healthcare delivery systems promote evidence-based practices and patient-centered outcomes research in delivering care.
Print | AJMC Printing...