Currently Viewing:
Supplements The Aligning Forces for Quality Initiative: Summative Findings and Lessons Learned From Efforts to Improve Healthcare Quality at the Community Level
The Aligning Forces for Quality Initiative: Background and Evolution From 2005 to 2015
Dennis P. Scanlon, PhD; Jeff Beich, PhD; Brigitt Leitzell, MS; Bethany W. Shaw, MHA; Jeffrey A. Alexander, PhD; Jon B. Christianson, PhD; Diane C. Farley, BA; Jessica Greene, PhD; Muriel Jean-Jacques,
Summative Evaluation Results and Lessons Learned From the Aligning Forces for Quality Program
Dennis P. Scanlon, PhD; Jeffrey A. Alexander, PhD; Megan McHugh, PhD; Jeff Beich, PhD; Jon B. Christianson, PhD; Jessica Greene, PhD; Muriel Jean-Jacques, MD, MAPP; Brigitt Leitzell, MS; Yunfeng Shi,
Currently Reading
The Longitudinal Impact of Aligning Forces for Quality on Measures of Population Health, Quality and Experience of Care, and Cost of Care
Yunfeng Shi, PhD; Dennis P. Scanlon, PhD; Raymond Kang, MA; Megan McHugh, PhD; Jessica Greene, PhD; Jon B. Christianson, PhD; Muriel Jean-Jacques, MD, MAPP; Yasmin Mahmud, MPH; and Jeffrey A. Alexande
Improving Care Delivery at the Community Level: An Examination of the AF4Q Legacy
Megan McHugh, PhD; Jillian B. Harvey, MPH, PhD; Jaime Hamil, MPH; and Dennis P. Scanlon, PhD
From Rhetoric to Reality: Consumer Engagement in 16 Multi-Stakeholder Alliances
Jessica Greene, PhD; Diane C. Farley, BA; Jon B. Christianson, PhD; Dennis P. Scanlon, PhD; and Yunfeng Shi, PhD
Lessons Learned About Advancing Healthcare Equity From the Aligning Forces for Quality Initiative
Muriel Jean-Jacques, MD, MAPP; Yasmin Mahmud, MPH; Jaime Hamil, MPH; Raymond Kang, MA; Philethea Duckett, MPA; and Juliet C. Yonek, MPH, PhD
Aligning Forces for Quality Multi-Stakeholder Healthcare Alliances: Do They Have a Sustainable Future
Jeffrey A. Alexander, PhD; Larry R. Hearld, PhD; Laura J. Wolf, MSW; and Jocelyn M. Vanderbrink, MHA
Evaluating a Complex, Multi-Site, Community-Based Program to Improve Healthcare Quality: The Summative Research Design for the Aligning Forces for Quality Initiative
Dennis P. Scanlon, PhD; Laura J. Wolf, MSW; Jeffrey A. Alexander, PhD; Jon B. Christianson, PhD; Jessica Greene, PhD; Muriel Jean-Jacques, MD, MAPP; Megan McHugh, PhD; Yunfeng Shi, PhD; Brigitt Leitze
Participating Faculty
eAppendix
Letter From Donald M. Berwick, MD, MPP, Guest Editor
Donald M. Berwick, MD, MPP
The View From Aligning Forces to a Culture of Health
Carolyn E. Miller, MSHP, MA, and Anne F. Weiss, MPP
Leading Multi-sector Collaboration: Lessons From the Aligning Forces for Quality National Program Office
Katherine O. Browne, MBA, MHA; Robert Graham, MD; and Bruce Siegel, MD, MPH
Healthcare Reform Post AF4Q: A National Network of Regional Collaboratives Continues Healthcare Reform From the Ground Up
Elizabeth Mitchell and Dianne Hasselman, MSPH

The Longitudinal Impact of Aligning Forces for Quality on Measures of Population Health, Quality and Experience of Care, and Cost of Care

Yunfeng Shi, PhD; Dennis P. Scanlon, PhD; Raymond Kang, MA; Megan McHugh, PhD; Jessica Greene, PhD; Jon B. Christianson, PhD; Muriel Jean-Jacques, MD, MAPP; Yasmin Mahmud, MPH; and Jeffrey A. Alexande
Objective: To summarize the results from the quantitative analyses conducted during the summative evaluation of the Aligning Forces for Quality (AF4Q) initiative.

Study Design: Longitudinal design using linear difference-in-difference (DD) regression models with fixed effects. Outcomes were selected based on the AF4Q program logic model and organized according to the categories of the Triple Aim: improving population health, improving quality and experience of care, and reducing the cost of care.

Data: Two primary data sources: the AF4Q Consumer Survey and the National Study of Physician Organizations (NSPO); and 4 secondary data sources: the Dartmouth Atlas Medicare claims database, the Truven Health MarketScan commercial claims database, the Behavioral Risk Factor Surveillance System (BRFSS), and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS).

Results: In total, 144 outcomes were analyzed, 27 were associated with improving population health, 87 were associated with improving care quality and experience, and 30 were associated with reducing the cost of care. Based on the estimated DD coefficients, there is no consistent evidence that AF4Q regions, over the life of the program, showed greater improvement in these measures compared with the rest of the United States. For less than 12% of outcomes (17/144), the AF4Q initiative was associated with a significant positive impact (P ≤.05), although the magnitude of the impact was often small. Among the remaining outcomes, with some exceptions, similarly improving trends were observed in both AF4Q and non-AF4Q areas over the period of intervention.

Conclusion and Policy and Practice Implications: Our quantitative findings, which suggest that the AF4Q initiative had less impact than expected, are potentially due to the numerous other efforts to improve healthcare across the United States, including regions outside the AF4Q program over the same period of time. The limited overall impact may also be due to the variability in the “dose” of the interventions across AF4Q regions. However, these results should not be interpreted as a conclusive statement about the AF4Q initiative. More nuanced discussions of the implementation of interventions in the specific AF4Q programmatic areas and their potential success (or lack thereof) in the participating communities are included in other articles in this supplement.

Am J Manag Care. 2016;22:S373-S381
After almost 10 years, the Robert Wood Johnson Foundation’s (RWJF’s) Aligning Forces for Quality (AF4Q) program, the largest privately funded community-based healthcare initiative to date, ended in April 2015.1 The AF4Q initiative’s design assumed that efforts to improve health and healthcare would be more effective when key stakeholders in a community collaborate to design, coordinate, and implement various interventions aimed at improving health system processes and outcomes.2 Between July 2006 and April 2010, 17 multi-stakeholder coalitions (alliances) in communities representing 12.5% of the US population were competitively selected as grantees; 16 were still part of the program when the AF4Q initiative ended. Over the life of the AF4Q program, RWJF dedicated $300 million in the form of direct grants to participating alliances; payments for technical assistance, program administration, and communication; and funds for an independent evaluation.

The design and implementation of the AF4Q initiative might be characterized by the program’s complexity and ambitious scope and the participating communities’ diverse geography and distinctive contextual factors, which included small rural areas such as Humboldt County in California; large metropolitan areas such as Boston; and 6 states. As discussed in greater detail in this supplement,1 the AF4Q interventions, to be implemented by regional multi-stakeholder alliances, were targeted at 5 primary programmatic areas: performance measurement and public reporting, consumer engagement, quality improvement, health equity, and health system payment reform. According to RWJF’s goals for the AF4Q program and its theory of change,1 focused and planned alignment among these different intervention areas would enhance the effectiveness of the initiative in improving population health outcomes, creating more patient-centered care, and improving the value and efficiency in the use of healthcare resources.

As described by Scanlon et al in an online-only article3 from this supplement, the complexity, scope, and length of the AF4Q initiative created a unique challenge for designing a comprehensive evaluation. Based on the structure and theory of the program, a logic model was developed to provide conceptual links between the various aspects of the program and potentially affected outcomes. The logic model also informed the selection and measurement of outcomes to be studied.1 The evaluation of the AF4Q initiative employed a mixed-methods approach, and numerous articles were published using quantitative and/or qualitative methods to address specific research questions. Some of these studies focused on a particular programmatic area,4,5 while others addressed important issues across different programmatic areas.6 In this article, from a summative perspective, we present a unified empirical framework to examine the impact of the AF4Q initiative on a broad set of quantitatively measured outcomes, linked to the 3 important aims in healthcare delivery: improving population health, improving quality and experience of care, and reducing the cost of care.7

The Triple Aim was proposed by Berwick et al in 2008,7 2 years after the beginning of the AF4Q initiative; hence, it was not mentioned in the original program design. However, the vision and design of the AF4Q initiative was consistent with the pursuit of the Triple Aim.8 More specifically, a central component of the AF4Q program mission was the establishment of local multi-stakeholder alliances operating as the potential “integrators” in selected communities, which has been suggested as a key precondition to achieving progress in the Triple Aim.7 As the AF4Q program proceeded, the Triple Aim became increasingly recognized by some key participants as the ultimate goal of such an initiative. Equally important, the outcomes presented in this paper were selected at the outset of the evaluation based on the logic model and the targeted intervention areas in the AF4Q program and before the concept of the Triple Aim initially appeared in the literature. Nevertheless, in retrospect, these preselected quantitative outcomes fit nicely in the Triple Aim framework, which is therefore used to organize our discussion of the results.

Although the quantitative outcomes together may provide an informative and objective synopsis of how much overall progress occurred in the 16 communities regarding the 3 aims, the analysis in this paper was not meant to capture important contextual factors and qualitative components of the AF4Q program. A comprehensive assessment of the AF4Q initiative is presented in Scanlon et al3 and qualitative assessments of the specific AF4Q programmatic areas can be found in several other papers in this supplement.9-13

The Quantitative Approach in the Summative Evaluation

Overview

The design of the quantitative component of the AF4Q summative evaluation followed 3 principles. First, to avoid data mining, the outcomes and analyses were conceptually driven and determined ex ante based on the program logic model. Therefore, these outcomes reflected the commitment of the investigators to the empirical aspect of the evaluation, a practice that has become increasingly important in large-scale evaluation research.14 Second, all outcomes were tracked longitudinally in the AF4Q regions as well as in other regions included as the comparison sample. Finally, to the extent allowed by the different data sets used in the evaluation, we maintained a consistent difference-in-difference (DD) modeling approach across all outcomes. In total, 144 outcomes were measured and analyzed using 6 different data sources.

Selection of Outcomes

Guided by the AF4Q logic model, we selected quantitative outcomes with several key considerations. First, the whole set of outcomes collectively reflected the potential overall impact of the AF4Q initiative in key areas of health and healthcare delivery. Second, the selected outcomes could be measured quantitatively with a reasonable level of validity and reliability. Third, the outcomes were reasonably well connected to the main programmatic areas, with priorities given to the known focal areas of the alliances’ interventional activities (eg, outcomes related to diabetes and chronic illness). Finally, each AF4Q alliance was required to produce and publicly release reports comparing provider quality in the region. The measures adopted in these public quality reports, often considered important indicators for healthcare quality in the participating communities, were used as references in selecting the outcomes to be evaluated. However, we did not use the actual scores of the quality measures publicly reported by the AF4Q alliances in our analysis because the reported measures, patient populations covered, number of providers included, and the starting time of reporting all varied significantly across the AF4Q alliance communities.

Among the 144 selected outcomes, 27 were associated with improving population health, 87 were associated with improving care quality and experience, and 30 were associated with reducing the cost of care. Each outcome was also categorized based on whether it might exhibit impact in the intermediate or long term, according to the logic model, and was linked to the relevant AF4Q programmatic area(s). The list of all selected outcomes is included in the online eAppendix. To understand the potential change in health equity among different racial and ethnic groups, another programmatic area of the AF4Q initiative,12 30 of the 144 outcomes were selected for further analysis (details available upon request).

Data

Two primary data sets collected for this evaluation—the AF4Q Consumer Survey15 and the National Study of Physician Organizations (NSPO)16—and 4 secondary data sources—the Dartmouth Atlas Medicare claims database,17 the Truven Health MarketScan commercial claims database,18 the Behavioral Risk Factor Surveillance System (BRFSS),19 and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS)20—were used to operationalize the 144 outcomes in this analysis. The AF4Q Consumer Survey was a longitudinal survey for chronically ill adults in the 16 AF4Q communities and was completed in 2 waves (August 2008 and November 2012). A comparison sample from the rest of the country was included. The NSPO was also completed in 2 waves (March 2009 and November 2013) and included representative samples of physician organizations from each AF4Q region. A third wave of the AF4Q Consumer Survey and the NSPO was originally planned, but RWJF decided in 2014 not to fund a third wave of either survey. The Dartmouth Atlas data provide zip code-level patient information aggregated from Medicare claims. The MarketScan data include individual claims from major commercial health plans (Humboldt County, California, was excluded from the AF4Q sample for outcomes using the MarketScan data, which did not allow us to identify single rural counties). The BRFSS is an annual telephone survey conducted by the CDC using repeated cross-sectional samples of individual patients with county identifiers. The HCAHPS annually surveys discharged patients about their recent experience in hospital care. HCAHPS data used in this study were aggregated at the hospital level and did not contain any hospital or patient characteristics. Table 1 summarizes the main features of the 6 data sets, including the target populations, the units of observation, and the structure and range of the data.

Empirical Strategy

We adopted a unified linear DD approach21 to analyzing all of the included outcomes, controlling for fixed effects at the most disaggregate unit allowed by the corresponding data. The baseline regression equation is specified as the following:



 
Copyright AJMC 2006-2018 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
x
Welcome the the new and improved AJMC.com, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up
×

Sign In

Not a member? Sign up now!