Currently Viewing:
Supplements The Aligning Forces for Quality Initiative: Summative Findings and Lessons Learned From Efforts to I
The Aligning Forces for Quality Initiative: Background and Evolution From 2005 to 2015
Dennis P. Scanlon, PhD; Jeff Beich, PhD; Brigitt Leitzell, MS; Bethany W. Shaw, MHA; Jeffrey A. Alexander, PhD; Jon B. Christianson, PhD; Diane C. Farley, BA; Jessica Greene, PhD; Muriel Jean-Jacques,
Currently Reading
Summative Evaluation Results and Lessons Learned From the Aligning Forces for Quality Program
Dennis P. Scanlon, PhD; Jeffrey A. Alexander, PhD; Megan McHugh, PhD; Jeff Beich, PhD; Jon B. Christianson, PhD; Jessica Greene, PhD; Muriel Jean-Jacques, MD, MAPP; Brigitt Leitzell, MS; Yunfeng Shi,
Reporting Provider Performance: What Can Be Learned From the Experience of Multi-Stakeholder Community Coalitions?
Jon B. Christianson, PhD; Bethany W. Shaw, MHA; Jessica Greene, PhD; and Dennis P. Scanlon, PhD
Improving Care Delivery at the Community Level: An Examination of the AF4Q Legacy
Megan McHugh, PhD; Jillian B. Harvey, MPH, PhD; Jaime Hamil, MPH; and Dennis P. Scanlon, PhD
From Rhetoric to Reality: Consumer Engagement in 16 Multi-Stakeholder Alliances
Jessica Greene, PhD; Diane C. Farley, BA; Jon B. Christianson, PhD; Dennis P. Scanlon, PhD; and Yunfeng Shi, PhD
Lessons Learned About Advancing Healthcare Equity From the Aligning Forces for Quality Initiative
Muriel Jean-Jacques, MD, MAPP; Yasmin Mahmud, MPH; Jaime Hamil, MPH; Raymond Kang, MA; Philethea Duckett, MPA; and Juliet C. Yonek, MPH, PhD
Aligning Forces for Quality Multi-Stakeholder Healthcare Alliances: Do They Have a Sustainable Future
Jeffrey A. Alexander, PhD; Larry R. Hearld, PhD; Laura J. Wolf, MSW; and Jocelyn M. Vanderbrink, MHA
Evaluating a Complex, Multi-Site, Community-Based Program to Improve Healthcare Quality: The Summative Research Design for the Aligning Forces for Quality Initiative
Dennis P. Scanlon, PhD; Laura J. Wolf, MSW; Jeffrey A. Alexander, PhD; Jon B. Christianson, PhD; Jessica Greene, PhD; Muriel Jean-Jacques, MD, MAPP; Megan McHugh, PhD; Yunfeng Shi, PhD; Brigitt Leitze
Participating Faculty
eAppendix
Letter From Donald M. Berwick, MD, MPP, Guest Editor
Donald M. Berwick, MD, MPP
The View From Aligning Forces to a Culture of Health
Carolyn E. Miller, MSHP, MA, and Anne F. Weiss, MPP
Leading Multi-sector Collaboration: Lessons From the Aligning Forces for Quality National Program Office
Katherine O. Browne, MBA, MHA; Robert Graham, MD; and Bruce Siegel, MD, MPH
Healthcare Reform Post AF4Q: A National Network of Regional Collaboratives Continues Healthcare Reform From the Ground Up
Elizabeth Mitchell and Dianne Hasselman, MSPH

Summative Evaluation Results and Lessons Learned From the Aligning Forces for Quality Program

Dennis P. Scanlon, PhD; Jeffrey A. Alexander, PhD; Megan McHugh, PhD; Jeff Beich, PhD; Jon B. Christianson, PhD; Jessica Greene, PhD; Muriel Jean-Jacques, MD, MAPP; Brigitt Leitzell, MS; Yunfeng Shi,
Objective: To report summative evaluation results from the Aligning Forces for Quality (AF4Q) initiative, the Robert Wood Johnson Foundation’s (RWJF’s) signature effort to improve quality of care from 2005 to 2015.

Methods: This was a longitudinal mixed methods program evaluation (ie, multiphase triangulated evaluation) of 16 grantee “alliances” from across the country, funded by RWJF as part of the AF4Q initiative. Grantees were selected in a nonexperimental manner and were charged with deploying interventions in 5 main programmatic areas to improve health and healthcare in their communities.

Results: Except for a small proportion of outcomes, there were no major differences in the rate of longitudinal improvement in AF4Q communities, compared with control communities, on quantitative outcomes related to the Triple Aim. Although the majority of the measures improved in both AF4Q and non-AF4Q communities, there were some exceptions to this improving trend, most noticeably in the cost of care and population health. There was also considerable heterogeneity across communities in terms of programmatic areas and the scale and scope of interventions in these areas. Although a number of AF4Q alliances implemented robust interventions in specific areas, often advancing strategies useful for others in the field, no AF4Q alliance pursued and aligned all 5 AF4Q programmatic areas in a robust way. In addition, whereas all alliances were able to garner the participation of multiple stakeholders initially, sustaining this participation and securing new sources of funding after RWJF support ended proved challenging for many alliances.

Conclusion and Policy and Practice Implications: While the AF4Q program did not attain the ambitious community-level changes predicted by its sponsor at the program’s outset, it did produce pockets of success on some dimensions for particular alliances. A number of factors explain the less-than-expected impact of the AF4Q initiative on community health and the observed variation in alliance sustainability and intervention strength. These include differing acceptance of the AF4Q initiative’s theory of change, variation in the experience and capacity of the alliance communities selected for the program, differences in alliances’ local healthcare market context, and the changing programmatic requirements for alliances participating in the AF4Q initiative. The variation in AF4Q program outcomes offers important lessons for those engaged in regional health improvement work.

Am J Manag Care. 2016;22:S360-S372
Aligning Forces for Quality (AF4Q) was the Robert Wood Johnson Foundation’s (RWJF’s) large and ambitious initiative focused on reforming local health systems in 17 communities by 2015. (RWJF selected 17 communities to participate in the AF4Q initiative over 3 phases of the program. However, one of the grantee communities, Central Indiana, was dropped from the AF4Q program. Thus, we refer to 16 communities for the remainder of the text.) The AF4Q initiative was launched in 2006, before passage of the Affordable Care Act (ACA) in 2010, and was built with community-based multi-stakeholder alliances serving as the “backbone organizations” in what many now describe as “collective-impact” approaches to addressing complex social problems.1 The program included multiple interventions and goals to significantly improve community health, which were developed and revised throughout its nearly decade-long lifespan.

Because of RWJF’s ambitious goals for the AF4Q initiative and the importance of this major community-based health reform effort, there are many stakeholders (eg, policy makers; philanthropic organizations; community-based coalitions, alliances, and multi-stakeholder groups; healthcare providers; healthcare payers; patients and patient advocates; and researchers and evaluators, among others) interested in knowing the impact of this program and whether the initiative as a whole, or specific parts of it, were successful.

Serving as an entryway into the independent, external evaluation team’s detailed findings, this article provides an overview of how we arrived at our approach for studying the impact of the AF4Q initiative (development of the evaluation research design and identification of RWJF’s theory of change for the program), as well as a summary of findings from each component of our assessment of the success of the program. Rather than define the success of the AF4Q initiative based on a single metric, we have approached assessment of the AF4Q program’s success as multidimensional. Findings are presented from multiple levels and perspectives, with the goal of allowing stakeholders to focus on the dimensions of success that are most meaningful to their particular contexts and needs. Barriers to success and key lessons learned from the AF4Q experience are also discussed.

Other articles in this supplement provide greater detail on the background and evolution of the program, the research design of the evaluation, the specific summative assessments of success in each of AF4Q’s 5 main programmatic areas (measurement and reporting of provider performance, consumer engagement [CE], quality improvement [QI], equity, and payment reform), an assessment of the community-level outcomes of the program, and an assessment of how the AF4Q multi-stakeholder alliances were positioned for the future when the program ended in 2015.

AF4Q Evaluation Research Design and Methods

Our evaluation research design was organized within an overarching logic model of the program (described below) and used a multiphase design (formally referred to as a “methodological triangulated design”2), which included sub-projects with independent methodological integrity in each of the AF4Q initiative’s 5 main programmatic areas along with an additional sub-project that focused on the organization and governance of the AF4Q alliances.3 For this nearly decade-long, complex program, we also employed elements of the Realistic Evaluation approach advanced by Pawson and Tilley, which states that “Programs work (have successful ‘outcomes’) only in so far as they introduce the appropriate ideas and opportunities (‘mechanisms’) to groups in the appropriate social and cultural conditions (‘contexts’).”4

Our evaluation, which focused on the development of an empirically based assessment of the final outcomes of the program, consisted of 3 phases. The first was a foundational phase in which the evaluation program logic model was developed, key research questions were identified, and data collection was put into motion. The second phase included systematic monitoring and measurement of program interventions and program and environmental changes. These changes included progress on intermediate outcomes and tracking AF4Q community involvement in the myriad of regionally focused healthcare improvement programs that overlapped during the AF4Q program period (eg, Chartered Value Exchange project, the Beacon Community Program, the Health Information Technology Extension Program, CMS Innovation Center programs).5 The third was a summative phase in which we, informed by the formative work, moved to answering the following 2 research questions: (1) Was the AF4Q program successful? (2) What lessons were learned from the AF4Q initiative that can inform those interested in improving local healthcare systems and the health of populations residing within these communities?

The AF4Q Initiative’s Theory of Change

The following statements describing the theory of change of the AF4Q initiative were published by RWJF leadership early in the implementation of the program:

“We launched the first phase of Aligning Forces for Quality: The Regional Market Project, a long-term, multi-million dollar commitment, to help a number of test communities re-weave the fabric of their own local health care system into a stronger, more resilient, higher-quality tapestry of care across its fullest continuum. We call it AF4Q. This is not piecemeal, incremental, short-term (and unsuccessful) health system reform as usual. It has no politics or partisanship of its own. If it did, it wouldn’t work and we wouldn’t do it. Rather, it is an unprecedented regionally determined clinical, social and economic market realignment that calls upon enlightened and aspirational local leadership, intentional collaboration, reliance on evidence-based action, public reporting and accountability, and public participation in deciding how quality health care is delivered to the community. AF4Q is a first-of-its-kind effort that is as much a call to community action as it is a potent formula to bring the best possible medical care and peace of mind to as many people and their families as possible.”6

“In June 2008 the Robert Wood Johnson Foundation (RWJF) launched phase II of Aligning Forces for Quality, a long-term, $300 million initial commitment to help up to twenty geographically, economically, and demographically diverse communities reweave the fabric of their health care systems to be stronger, more resilient, and of higher quality across the full continuum of care.”

“The RWJF’s objective is to help the Aligning Forces communities improve the quality of care for everyone in these communities by 2015. If these communities, with widely varying provider and payer systems, racially and ethnically diverse populations, and differing chronic disease rates, can improve care with this concerted focus, then improving quality nationwide is achievable.”7
 

As the quotes illustrate, the AF4Q initiative was designed to be a far-reaching and ambitious initiative with multiple levels and types of outcomes. Additionally, RWJF added additional components and modified expectations for the established program areas over the course of implementation. Although each of the participating alliances were required to implement activities in all 5 programmatic areas, there was variability in how they approached the work and how much time and resources the alliances dedicated to them.

To better conceptualize the numerous dimensions of success and the mechanisms for achieving those outcomes in RWJF’s theory of change, we developed an AF4Q logic model. We did this through careful review of RWJF documentation about the program and conversations with RWJF leaders. Additionally, the AF4Q logic model was updated over time to reflect changes in the program. It is described in full detail in another article in this supplement which discusses the background, history, and evolution of the AF4Q program.8

The AF4Q logic model can be thought of as containing 4 main components. The first component relates to the creation and development of a functioning multi-stakeholder alliance that identifies priorities for improving health and healthcare in the community and sets strategy for how to accomplish these goals. The second component involves selecting and implementing interventions in the 5 AF4Q programmatic areas and attempting to align these interventions so they complement one another; as specified in the program name, alignment of the various program interventions was originally envisioned to be a key differentiator in the AF4Q initiative. The third component involves measuring the intermediate and long-term community-level outcomes that RWJF hypothesized would stem from the aforementioned interventions, with intermediate outcomes focused more on short-term process changes while long-term outcomes focused on changes in communitywide health and cost outcomes. The fourth component relates to sustaining the capacity for improvement of the health system at the community level and “scaling up” the overall effort to reach the entire population, especially in the context of all of the other changes occurring in the broader healthcare system. Related to this fourth component, the degree to which AF4Q alliances became models for reform for others across the country, an explicit goal of RWJF, is another important dimension to consider when evaluating the success of the AF4Q initiative.

To answer the first research question (Was AF4Q successful?), we organized the discussion by starting with the most global components of the logic model—communitywide outcomes, sustainability, and opinions about the degree to which the AF4Q initiative provided models for other communities. The discussion of success later focuses on the implementation and alignment of specific program interventions. Finally, we provide some reasons for the pattern of results that were observed.

Was the AF4Q Program Successful?

Communitywide Outcomes, Sustainability, and Thought Leader Opinions

Intermediate and Long-Term Communitywide Outcomes—Little Impact on Outcomes Relative to Non-AF4Q Communities. The most ambitious aspiration for the AF4Q initiative was that it would result in improvements in community (population) health and healthcare quality measures and yield more value for resources spent on healthcare services within the participating AF4Q communities. To study whether this result was achieved, we selected a broad set of measures (144 in total) early in the evaluation and then grouped them into the 3 Triple Aim categories of better health, better care, and lower costs that we could follow throughout the life of the AF4Q program. We monitored these same measures in both AF4Q and non-AF4Q communities because it enabled us to assess whether any improvements (or declines) in measures in AF4Q communities could more assuredly be attributed to participation in the program, as opposed to a more general trend in outcome measure improvement happening across the country. Details about the selection of measures, the data sources for these measures, and our analyses, which utilized a difference-in-differences approach, are outlined in the article by Shi et al in this supplement.9

 
Copyright AJMC 2006-2017 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
x
Welcome the the new and improved AJMC.com, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up
×

Sign In

Not a member? Sign up now!