The Aligning Forces for Quality Initiative: Summative Findings and Lessons Learned From Efforts to Improve Healthcare Quality at the Community Level, 

Articles

EAPPENDIX

eAppendix

Dr Berwick is president emeritus and a senior fellow at the Institute for Healthcare Improvement, an organization that he co-founded and led as president and CEO for 18 years. He is one of the nation’s leading authorities on healthcare quality and improvement.

In July 2010, President Barack Obama appointed Dr Berwick as administrator of CMS, a position he held until December 2011. A pediatrician by background, Dr Berwick has served as a clinical professor of pediatrics and healthcare policy at Harvard Medical School, a professor of health policy and management at the Harvard School of Public Health, and as a staff member at Boston Children’s Hospital, Massachusetts General Hospital, and Brigham and Women’s Hospital. He has also served as vice chair of the US Preventive Services Task Force, the first independent member of the board of trustees of the American Hospital Association, and chair of the National Advisory Council of the Agency for Healthcare Research and Quality. An elected member of the Institute of Medicine (IOM), Dr Berwick served 2 terms on the IOM’s governing council and was a member of the IOM’s Board on Global Health. He served on President Bill Clinton’s Advisory Commission on Consumer Protection and Quality in the Health Care Industry.

Dr Berwick is a recipient of numerous awards, including the 1999 Joint Commission Ernest Amory Codman Award, the 2002 American Hospital Association Award of Honor, the 2006 John M. Eisenberg Patient Safety and Quality Award for Individual Achievement from the National Quality Forum and the Joint Commission on Accreditation of Healthcare Organizations, the 2007 William B. Graham Prize for Health Services Research, the 2007 Heinz Award for Public Policy from the Heinz Family Foundation, the 2012 Gustav O. Lienhard Award from the IOM, and the 2013 Nathan Davis Award from the American Medical Association. In 2005, he was appointed an honorary knight commander of the British Empire by Queen Elizabeth II, the highest honor awarded by the United Kingdom to non-British subjects, in recognition of his work with the British National Health Service.

Dr Berwick is the author or co-author of more than 160 scientific articles and 6 books. He also currently serves as a lecturer in the Department of Health Care Policy at Harvard Medical School.

Network for Regional Healthcare Improvement

Physician-Focused Payment Model Technical Advisory Committee

Faculty Disclosures—Perspective Articles

The Aligning Forces for Quality (AF4Q) evaluation is funded by a grant from the Robert Wood Johnson Foundation (RWJF).

These faculty report relationships with the following organizations:

James A. Hamilton Chair in Health Policy and Management

AUTHOR INDEX

Participating Faculty

 This analysis assessed the evolution of public reporting of provider performance in Aligning Forces for Quality (AF4Q) alliances, contrasted alliances that stopped reporting with those that plan to continue, and drew insights from alliance public reporting efforts for the national transparency movement.

 Combined with document review, qualitative research methods were used to analyze interview data collected, over a nearly 10-year period, from the 16 participating alliances.

 AF4Q alliances made their greatest contributions to provider transparency in reporting ambulatory quality and patient experience measures. However, reporting ambulatory cost/efficiency/utilization measures was more challenging for alliances. Alliances contributed the least with respect to measures of inpatient performance. Six alliances ceased reporting at the end of the AF4Q program because of their inability to develop stable funding sources and overcome stakeholder skepticism about the value of public reporting. Insights provided by alliance leaders included the need to: focus on provider, rather than consumer, responses to public reports as the most likely avenue for improving quality; address the challenge of funding the reporting infrastructure from the beginning; explore collaborations with other entities to increase public reporting efficiency; and develop a strategy for responding to efforts at the national level to increase the availability of information

The AF4Q initiative demonstrated that a wide variety of voluntary stakeholder coalitions could develop public reports with financial and technical support. However, the contents of these reports varied considerably, reflecting differences in local environments and alliance strategies. The challenges faced by alliances to maintain their reporting efforts were substantial, and not all alliances chose to report. Nevertheless, there are potential roles for alliances going forward in contributing to the national transparency movement.

In this article, we examine the efforts of voluntary stakeholder coalitions (ie, alliances) to measure and publicly report provider performance as part of the Robert Wood Johnson Foundation’s (RWJF’s) Aligning Forces for Quality (AF4Q) initiative. (Details regarding these alliances and the communities they served are outlined in the article by Scanlon et al located in this supplement.

) Public reporting of provider performance was a key component of the overall strategy of the AF4Q initiative to improve the quality of care in alliance communities (see logic model in 

). Transparent quality measures could encourage consumers to choose higher-quality providers, enhance patient interactions with providers, and stimulate providers to undertake quality improvement activities to avoid losing patients and the stigma that might be attached to poor performance on quality measures. Health plans could use these measures to develop benefit designs that distinguish higher-quality from lower-quality providers (possibly rewarding consumers for choosing higher-quality providers) and implement pay-for-performance programs. This view of the possible benefits of public reporting was similar to that expressed by national advocates of greater provider performance transparency.

Consistent with this emphasis, alliances were selected to participate in the AF4Q program in part because they had experience with, or expressed a willingness to implement, public reporting.

 However, our analysis (further detailed below) found substantial variation in the performances of alliances with respect to AF4Q public reporting. Some met the reporting goals and timelines established by the AF4Q program and planned to continue reporting after the program’s conclusion, others were less successful, and some stopped reporting entirely once the AF4Q program funding ended. Overall, in a separate analysis, we found that consumers in alliance communities had access to more provider performance information than available to residents of comparison communities

 and this remained true throughout the AF4Q initiative.

To place the AF4Q program public reporting efforts in context, we begin by discussing national performance measurement and public reporting efforts prior to, during, and at the end of the AF4Q initiative. We summarize expectations for measurement and public reporting from the AF4Q program and how they changed over the course of the initiative. After describing our methods for data collection and analysis, we address 3 questions: (1) Across AF4Q communities, how did public reporting evolve and what factors drove that evolution? (2) At the end of the AF4Q program, what distinguished alliances that continued to report from those that stopped reporting? (3) What insights can be drawn from the alliance public reporting experience to inform national efforts to increase provider performance transparency?

Reporting health plan quality measures began in the early 1990s

 under the auspices of the National Committee for Quality Assurance (NCQA). The federal Health Care Financing Administration (now CMS) and several states (eg, New York, Pennsylvania, and California) began reporting hospital quality measures during this time period, as well. By 2000, large employers were encouraging the reporting of physician performance as part of an overall healthcare reform strategy.

 Many health plans responded by reporting a limited number of provider quality measures to their members, or by “tiering” providers based on quality measures, but this information usually was not available to the general public. Private firms also entered the reporting arena (eg, Healthgrades and WebMD), with consumers typically paying for full access to their provider performance measures.

The National Quality Forum (NQF) was established in 1999 as a public—private partnership to “create a foundation for consistent data reporting and collection.”

 Incorporating measures developed by NCQA or endorsed by NQF subsequently became the “gold standard” for public reporting. In 2001, under the auspices of the Leapfrog Group, large employers began to publish information on hospital patient safety practices, using data voluntarily submitted by some hospitals. The Ambulatory Care Quality Alliance (the American Academy of Family Physicians, American College of Physicians, America’s Health Insurance Plans, and the Agency for Healthcare Research and Quality), established in 2004, produced a “starter set” of physician quality measures, which it pilot-tested in 2006. In the public sector, Hospital Compare was created by CMS in 2002, and in 2005 the first process of care measures were displayed on the Hospital Compare website, with patient experience measures added soon after.

Clearly, by the time the first alliances were selected for the AF4Q program in 2006, there was considerable momentum for reporting provider performance, especially quality measures. Subsequently, several additional public-sector reporting efforts were initiated, beginning with the establishment of the Chartered Value Exchange (CVE) program by the Bush administration in 2008. Under this program, community organizations could apply for CVE designation, which entailed a commitment to publish provider quality information. In return, CVEs were to receive access to Medicare performance measurement results, along with technical assistance through a peer-learning network. Of the 24 organizations that received the CVE designation by 2012, 11 were AF4Q alliances.

Support for public reporting expanded under the Obama administration’s Affordable Care Act (ACA), which required CMS to share Medicare data with “qualified” local entities for use in reporting provider performance.

 In addition, advocacy groups encouraged the reporting of physician quality measures on ACA health insurance exchanges. Meanwhile, Medicare’s Physician Compare effort provided data on individual physician characteristics: in 2015, patient assessments of care and clinical quality measures became available for practices and groups. This effort also gave consumers access to information on whether individual providers or medical groups participated in various quality programs, including the Physician Quality Reporting System, the Million Hearts initiative, and the Electronic Health Records Incentive Program.

 In addition to its focus on physicians, CMS initiated public reporting for skilled nursing facilities and home healthcare.

Alongside these federal efforts, 43 states have developed, or plan to develop, all-payer claims databases, providing a resource that could be used in producing provider performance reports.

 Subsequent to the establishment of the AF4Q program, private-sector purchasers increased their support for the reporting of measures of cost and efficiency, in addition to quality measures,

 with the hope that consumers would consider “value” (defined as quality relative to dollars spent) when making their choices and that providers would be encouraged to compete based on the value of the services they provided.

Two of the first 4 alliances selected to participate in AF4Q were already publicly reporting provider performance, and a third was in the process of measure construction. However, there were still relatively few community coalition public reporting efforts nationally at the onset of the AF4Q program, although some coalitions were actively involved in Leapfrog reporting efforts.

 There were several aspects of locally produced reports that appeared promising (ie, adding value to national reporting efforts) and deserving of support in the AF4Q initiative. Locally produced reports could increase the number of different sources of performance information available in the community, raising the likelihood that consumers would become aware of provider performance measures. It was hoped that the reports would draw significant attention, because they were “locally developed,” and that providers would view them as credible, especially when they were involved in measure selection or development. Finally, to the degree that local reporting efforts were guided by input from community stakeholders, their measures might be more salient to community residents than those in national reports.

When they joined the AF4Q program, alliances were charged with publicly reporting (within 3 years) measures of ambulatory quality in the treatment of chronic conditions for at least 50% of primary care physicians in their communities. Alliance leaders felt that their continued funding could very well depend on complying. Almost all alliances reached this goal, although report content varied considerably.

 Subsequently, expectations for alliances were expanded to include reporting patient experience measures for ambulatory care, hospital quality and patient experience measures, and measures of resource use, charge, price, cost and/or efficiency in inpatient and outpatient settings. Alliances were encouraged to make their reports “consumer -friendly” and to pursue different methods of disseminating reports. By the end of the AF4Q program in 2015, the alliances were expected to produce public reports that facilitated consumer selection of “high-value” providers.

In summary, the reporting goals for AF4Q alliances were very ambitious. The hope was that alliances would play a leading role in the emerging transparency movement by providing “models” for public reporting that could be adopted in other communities. During the nearly 10-year AF4Q initiative, the number of NQF and NCQA performance measures of all types expanded considerably, which aided alliances in selecting and constructing measures for their reports. In addition to funding, the AF4Q program provided alliances with technical assistance to support their reporting efforts.

We used 3 data sources in our analyses. We tracked the contents of alliance reports throughout the AF4Q program and constructed a longitudinal dataset for each alliance that contained information on the types of measures reported, start and stop dates for reporting of measures, level of reporting (eg, medical group or medical practice), data sources used to construct each measure (eg, claims, medical records, surveys), and frequency with which measures were reported (eg, annually, biannually). Using this dataset, we examined how reporting evolved across alliances and over time.

Beginning in 2006, and continuing through 2016, the evaluation team conducted 1100 semi-structured in-person and telephone interviews with alliance leaders, staff, and community stakeholders, with most addressing issues related to public reporting. This included 4 rounds of site visits to AF4Q communities, 10 rounds of telephone interviews with alliance leaders between 2007 and 2014, and 2 rounds of telephone interviews between 2012 and 2014 with alliance leaders specifically charged with managing pubic reporting efforts. The semi-structured interviews provided information about alliance commitment to reporting, alliance reporting strategies, and the challenges alliances encountered. Latter rounds of interviews also addressed alliance decisions about whether or not to sustain reporting after the AF4Q program and provided insights regarding how multi-stakeholder coalitions might contribute most effectively to national provider transparency efforts. Telephone interviews also were conducted with key national thought leaders in healthcare to elicit their views on the impact of alliance reporting efforts. (The process for collecting and analyzing these interview data is described in detail in the summative research design article, which is available online at www.ajmc.com.

Our third source of data consisted of AF4Q program documents, including (1) information on guidelines and performance targets established by RWJF and the AF4Q’s national program office, (2) funding proposals submitted by alliances throughout the course of the AF4Q program, and (3) progress summaries that alliances submitted to the national program office every 4 months beginning in 2011.

Evolution of Public Reporting Across Alliances: Achievements and Challenges 

Of the 16 alliances, 5 reported provider performance (ambulatory or inpatient) prior to entering the AF4Q program, and 2 more were engaged in measure selection and development, with the intent of reporting; the remaining alliances began the public reporting process after joining the AF4Q initiative. Given their different starting points, it is not surprising that strategies for reporting, and the contents of reports, varied across alliances. Reports also evolved over time in response to new AF4Q program expectations, fluctuating stakeholder support, and experience gained with reporting and its consequences.

As in previous work, we view alliance report production as beginning with measure selection and continuing through measure construction and dissemination, with alliances revisiting their decisions during the AF4Q program (

). We extend findings from prior work, which included alliance reporting activities from 2006 to 2012,

 by addressing the latter part of the AF4Q initiative (2012-2015; see tables in 

). In the first section, we focus on measure selection, construction, and dissemination, examining 3 types of measures (quality, patient experience, and cost/efficiency/overutilization) for ambulatory and inpatient care. We discuss the evolution in reporting for each type and the important factors influencing that evolution.

Selection and Construction of Ambulatory Measures 

 Consistent with AF4Q program expectations, by 2012, all alliances reported measures of quality of care for patients with chronic conditions.

 One respondent observed, “That’s what you had to do to get the grant.” But at this stage, the reports varied considerably in their scope and sophistication. For example, one alliance that reported measures prior to the AF4Q program included 24 process and intermediate outcome measures relating to vascular disease, diabetes, hypertension, and kidney disease in its report, while an alliance that joined the AF4Q program later, and had no prior reporting experience, reported 4 diabetes process measures. Overall, most of the alliances increased the number of publicly reported quality measures between 2012 and the end of the program, while 3 alliances reduced the number of measures and 2 stopped reporting altogether.

By 2012, all 16 alliances were using nationally endorsed measures in their reports, and in that year, 6 of the alliances began reporting composite performance measures of varying types, in addition to disaggregate measures. These approaches remained unchanged through 2015. Alliances used claims data and medical records data to construct measures, with no clear trend toward more alliances using one source or the other during the nearly decade-long initiative. In 2015, 8 alliances relied on administrative data, 5 used medical records, and 1 combined these data sources.

These were the data sources used by the same alliances in 2012, as well. In contrast, some alliances altered the level at which they constructed and reported measures. In 2012, 5 alliances reported quality measures only at the medical group or large physician organization level, while 10 reported at the practice level and 1 at the individual level. As of 2015, of the 14 reporting alliances, 13 reported at the practice level and 1 reported only at the physician organization level. (The alliance that reported at the individual physician level stopped reporting altogether.)

Alliance efforts to include information from more patients and providers in measure construction were met with several obstacles. Because supplying data was costly, maintaining participation on the part of data suppliers (health plans or providers) often was challenging for alliances. When claims data were used for measure construction, alliances needed to integrate data provided voluntarily by health plans, Medicaid programs, and/or self-insured providers. Therefore, depending on the decisions of data suppliers, alliance reports at different points of time could reflect care provided to different subsets of community residents. One respondent noted that, by the end of the AF4Q program, health plans in the community had just grown “weary” of participating in alliance public reporting efforts, while another observed that a number of large self-insured employers in its community had declined to supply data for public reporting. Alliances that depended on providers to submit data from medical records for measure construction faced a similar challenge and, based on observations of interview respondents, the percentage of participating providers appeared to vary considerably across alliance communities. (Alliances typically did not track this number.) Consequently, there was variation in the ability of consumers within different alliance communities to compare performance across providers.

 Some alliance leaders felt that reporting ambulatory patient experience measures was a valuable extension of their reporting efforts. These measures were applicable to all consumers, in contrast to the much more limited applicability of quality measures for a specific chronic illness. In 2009, 2 alliances participated in a 

 pilot effort to test the feasibility of reporting patient experience at the individual physician level. Although neither alliance continued reporting beyond the pilot study, in 2010, the AF4Q program provided supplemental pilot funding to 3 other alliances to incorporate Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) questions into the existing survey instruments of a limited number of medical groups in their communities and to publicly report results. Two of the 3 alliances continued reporting after the pilot program. Stakeholder support for reporting patient experience grew throughout the AF4Q program, partly due to a national focus on improving the experience of patients in the healthcare system.

 By 2015, 12 of the 16 alliances had reported ambulatory patient experience measures at least once, and 5 reported on a regular basis. A leader of one of these 5 alliances said that ambulatory patient experience reporting may have been their primary contribution to provider transparency in the community.

The development and increasing use of CG-CAHPS measures at the national level made measure selection less of an obstacle to reporting. Despite this, fielding patient surveys to collect the data needed to construct these measures added to alliance reporting costs, and finding the necessary funds proved to be a significant obstacle for many alliances. Alliances responded by using different combinations of membership dues, grants, and state support. However, one respondent expressed hope for the future, observing that hospital acquisitions of physician practices made funding easier to obtain; hospitals possessed more resources to support the effort than did individual physician practices and had a history of gathering patient experience data.

Alliances were asked to report NQF-endorsed measures of resource use, charges, price and cost, or efficiency by July 2011, and to report for 50% of community providers by 2013. By 2012, 7 alliances had accomplished this objective by reporting measures of overutilization, with 1 also reporting a cost measure. The alliances typically reported 4 or fewer measures, with “avoidance of antibiotic treatment for bronchitis” and “appropriate imaging for low back pain” being the most common. By 2015, 9 alliances were reporting measures in this category, with the number and types of measures changing very little between 2012 and 2015, with 2 exceptions. One alliance added 3 cost measures (total cost of care for all patients, total cost of care for adults, and total cost of care for pediatrics) and a second added total cost of care for adults. Of these 9 alliances, 6 reported at the practice level, 2 at the group level, and 1 at both levels.

In general, alliances found that reporting ambulatory cost/efficiency performance was a daunting task, relative to reporting ambulatory quality measures. There were fewer nationally endorsed measures to draw from, and stakeholder support (especially support from health plans that sometimes viewed cost data as proprietary) was more difficult to achieve. One alliance respondent observed that “utilization and price [are] way closer to the pocketbooks of organizations and individuals than quality.” Constructing cost/efficiency measures was technically challenging, as well. Alliances that used claims data to report ambulatory quality measures employed the same source to construct cost/efficiency/overutilization measures where possible. (However, some faced contractual restrictions on use of claims data from health plans for this purpose.) None of the alliances that used only medical records data to report quality measures issued any reports containing ambulatory cost/efficiency/overutilization measures. Some tried to access state-based all-payer claims databases for this purpose, but faced limitations on who could use these data and for what purposes.

Beyond issues related to data access and the complexity of measure construction, arguably the largest obstacle to reporting in this area was a lack of agreement among alliance leadership and stakeholders concerning the usefulness of the effort. One alliance leader summarized this point of view: “I think the biggest challenge we have is that nobody seems to care. The consumer doesn’t use it, and nobody wants to pay for it.” In contrast, 1 alliance played a leadership role nationally in reporting an ambulatory “total cost of care” measure, while another felt that reporting cost/efficiency energized the alliance and placed it in a pivotal position to engage in healthcare reform in its community.

Selection and Construction of Inpatient Measures

 In 2008, the Foundation instructed the alliances to add inpatient quality measures to their public reports (and, subsequently, patient experience and cost/efficiency measures). Alliances could meet this new inpatient reporting requirement by using measures from Medicare’s Hospital Compare program; simply providing a link to the Hospital Compare website would not fulfill the hospital quality-reporting objective. Ten alliances reported Hospital Compare quality measures only, and an additional 4 alliances supplemented those measures with measures constructed from other data sources. Another alliance selected measures from other national reports and its state hospital association, and 1 chose not to report inpatient quality at all during this time period. Four alliances used the same technical assistance provider to help them populate their reports with Medicare inpatient measures. They reported an identical set of Hospital Compare measures that remained essentially unchanged throughout the AF4Q program. By the end of the initiative, only 9 alliances were still reporting inpatient quality measures.

Given alliance access to measures in Hospital Compare and other reports for inpatient quality reporting, measure selection was relatively straightforward and measure construction was not required (

). The primary challenge was convincing stakeholders that the effort was necessary. Because hospital quality reporting was already being done by Hospital Compare and other entities, stakeholders were skeptical that disseminating the same measures through alliance websites constituted a significant contribution to quality improvement in their communities.

 Fourteen of the 16 alliances reported inpatient experience, and all used Hospital Compare to meet this expectation. However, although these alliance reports added another point of access to inpatient experience measures for consumers, the measures did not expand the information already available. As with ambulatory patient experience measures, there was strong support among alliance stakeholders for reporting hospital patient experience and, as with measures of inpatient quality, importing the measures from Hospital Compare was relatively straightforward.

 Alliances were given considerable latitude regarding which measures would satisfy reporting requirements in this area. As they did with hospital quality, virtually all of the reporting alliances extracted measures from Hospital Compare. By 2015, 7 of the 16 alliances included these measures in their reports, with Medicare procedure costs and readmission rates being the most common. Measures drawn from Hospital Compare were based only on a subset of community residents—Medicare beneficiaries. Also, Medicare procedure costs are of limited usefulness for consumer decision making.

Once the alliances selected and constructed provider performance measures, they faced the challenge of how to effectively disseminate the measures to potential users, including clinicians, insurers, and consumers (

). All alliances posted their performance measures on websites that they maintained. Alliances sought out media coverage of the release of their reports and found it was an effective way to garner attention. This attention, however, was harder to generate on an ongoing basis as report releases became more routine.

Over time, RWJF placed more emphasis on dissemination of public reporting to consumers. In 2009, the Foundation informed alliances that to increase consumer awareness and use of comparative provider information, they should focus on making their websites more consumer friendly and engage in new outreach strategies. To monitor their success, alliances were required to report on the number of visits to their websites and page views related to their public reports.

 However, a separate analysis suggested that there were not substantial improvements in the consumer friendliness of alliance websites over the last 5 years of the AF4Q program. Alliances that were relatively new to reporting suggested that this was because they needed to devote their resources to report production.

 Also, some alliances remained unconvinced that consumers had interest in this type of information, citing provider resistance to the use of more intuitive, consumer-friendly measures of provider performance (eg, star ratings).

Disseminating reports to consumers was a challenge because few alliances had existing consumer constituencies for support. Instead, alliances sought to reach consumers through employer stakeholders, traditional media, and social media. The level of commitment to doing so was mixed, as not all alliances felt that the information in their reports was likely to be valued by consumers. One alliance leader said that healthcare professionals constituted the primary group interested in the reported measures; they were “the real consumer group that we’ve actually targeted.” In contrast, several alliances tried innovative approaches to make their reports more accessible to consumers. These included having a reporting website targeted only at consumers and including educational health messages to complement specific provider measures on the website.

The effort that received the greatest attention was an experimental collaboration between the Consumers Union (the policy and action division of 

) and 3 alliances to include their performance measures in local issues of 

 magazine. This generated a very large number of new visits to the alliance reporting websites, suggesting that there was a potential market for their performance data. Building on the Consumers Union pilot study, RWJF funded the DOCTOR Project, which provided guidance and funding for an additional 8 AF4Q alliances and 2 other stakeholder collaboratives to distribute their provider performance data through 

Summary of the Evolution of Alliance Public Reporting Efforts

AF4Q alliances increased consumer access to measures of ambulatory quality and patient experience; through their reporting and dissemination activities, they expanded both the type and amount of information available to consumers (

). In a separate analysis, we compared public reporting activities in alliance communities with a set of comparison sites. We found that consumers in alliance communities had access to more information overall, with the difference remaining relatively stable over time. The exception was patient experience, where the advantage enjoyed by consumers in alliance communities grew. For more information, see Christianson et al.

To date, alliances have made fewer contributions to ambulatory cost/efficiency/overutilization transparency. A small number of alliance leaders remain enthusiastic, but the challenges seem likely to discourage a majority of alliances from expanding their efforts in this area. Most alliance leaders and stakeholders viewed inpatient performance reporting as an area already “occupied” by other state or national organizations. Consequently, one respondent observed that the alliance “checked the boxes” on reporting inpatient measures so as to not jeopardize continued AF4Q funding; this is consistent with limited stakeholder support for more ambitious efforts. Report dissemination efforts varied widely, but, in general, alliances struggled to develop and implement effective means to reach consumers. Explanations included lack of stakeholder support, skepticism concerning whether consumers valued the information, and competing demands on alliance and stakeholder resources.

Sustaining Public Reporting by Alliances 

It would be reasonable to expect that alliances would sustain their public reporting efforts after the AF4Q program, given the high priority that RWJF assigned to public reporting and the funds and technical assistance the Foundation provided to support alliance reporting activities. At the conclusion of the AF4Q initiative, based on our interview data, 6 alliances seemed highly likely to continue their public reporting efforts, 4 appeared less certain, and 6 were not planning to construct any new reports. In this section, we discuss what distinguishes alliances that plan to continue their public reporting efforts from those that have stopped or plan to stop reporting.

Based on respondent interviews, 2 general themes emerged for most, but not all, of these 6 alliances. First, alliance leadership viewed public reporting as central to the alliance’s identity, or “brand.” Five of the 6 alliances were reporting provider performance, or were in the measure selection process, before they were chosen to participate in the AF4Q program. Their stakeholders were committed to public reporting, independent of AF4Q program expectations and support. One respondent observed that “it’s what we do.” Another person said, “We did it before, and we’ll do it after.” A second theme, consistent with the first, is that most of these alliances had relatively stable funding sources, unrelated to the AF4Q initiative, which could be used for reporting. One respondent characterized the AF4Q program as a “blip” in the alliance’s history and certainly not the only driver of the alliance’s priorities or funder of its activities.

Still, these 6 alliances varied in how they viewed their public reporting futures. In one alliance, leadership felt that the public reporting field in the community was “crowded.” Consequently, the alliance planned to reduce the number of measures reported and the frequency of reporting, focusing mostly on ambulatory patient experience. In another alliance, leaders were working to expand alliance funding from community employers, which they believed to be important for maintaining the alliance reporting infrastructure going forward. In a third alliance, inpatient measures were dropped from its report, as stakeholders felt that they were “least used” among its measures; this alliance was planning to focus future efforts on cost/efficiency/overutilization measurement, where its stakeholders saw greater value. A fourth alliance was playing a significant role in its community’s healthcare reform efforts; its leaders planned to reduce alliance emphasis on public reporting but continue it in the near term. These examples illustrate the significant variation among the alliances that anticipated continuing to report in how they envisioned their reporting futures.

Alliances Where Continued Reporting Appears Less Certain

Four alliances have less certain reporting futures, for various reasons. In one case, the alliance was moving reporting responsibilities to another community entity, raising questions about whether public reporting will be sustained after the transition. Second, in 3 instances where alliance leaders expressed a desire to continue reporting, there was not a clear plan to fund it. All were exploring the feasibility of raising funds from various sources, including health plans, local foundations, providers, and state grants. Two of these alliances lagged behind in updating existing reports and had experienced some erosion of stakeholder participation or funding. The leader of another alliance suggested that it would continue to measure and report privately as part of its quality improvement activities, but expressed skepticism that the public reporting of measures was beneficial to consumers.

Of the 6 alliances that discontinued public reporting, none were engaged in public reporting of provider performance prior to the AF4Q program. In most cases, their leaders and stakeholders initially were skeptical that reporting would lead consumers to choose higher-quality providers. This was true for alliances located in rural areas, where physician practices were full and there was concern about maintaining an adequate physician supply; however, similar sentiments were expressed by respondents associated with alliances in urban settings. Stakeholders saw more promise in devoting community time and resources to other endeavors (such as supporting provider quality improvement activities or identifying and publicizing pressing local healthcare issues), rather than continuing their public reporting efforts.

ARTICLES

Reporting Provider Performance: What Can Be Learned From the Experience of Multi-Stakeholder Community Coalitions?

 A key component of the Aligning Forces for Quality (AF4Q) program was engaging consumers in their health and healthcare. We examined the extent to which the alliances embraced 4 areas of consumer engagement: self-management, consumer friendliness of reports of healthcare provider quality, involvement of consumers in alliance governance, and the integration of consumers into quality improvement teams.

We used a largely qualitative approach. The evaluation team conducted 1100 in-depth interviews with alliance stakeholders. Two authors reviewed the consumer engagement data for each alliance to assess its level of embrace in the 4 consumer engagement areas. For consumer friendliness of public reporting websites, we also assessed alliance public reports for reading level, technical language, and evaluable displays. Population-level effects were also examined for self-management and public reporting.

 Consumer engagement was new to most alliances, and few had staff with consumer engagement expertise or existing consumer constituencies. For each area of consumer engagement, some alliances enthusiastically embraced the work, other alliances made a concerted but limited effort to develop programs, and a third group of alliances did the minimum work required. Integrating consumers into governance was the area most often embraced, followed by making public reports consumer friendly. Two alliances strongly embraced both self-management and integrating patients into quality improvement efforts. The AF4Q program did not have greater population level effects from self-management or public reporting than were those observed in a national comparison sample.

 The AF4Q program sparked a few alliances to develop robust consumer engagement programming, while most alliances tried consumer engagement efforts for the first time and developed an appreciation for integrating consumer perspectives into their work.

The concept of consumer engagement has received a lot of attention in recent years, being described as “the blockbuster drug of the century” and a key factor for achieving the Triple Aim.

 The idea that consumers can help improve the efficiency of healthcare delivery by being more involved in their health and making more informed choices about treatments and providers has widespread, bipartisan appeal.

 However, what exactly is meant by the term has been questioned by many, including Munro who wrote: “How we define it will determine whether it’s truly a miracle drug—or just another variant of age-old snake oil.”

Because of the ambiguity in definition, 2 recent papers have sought to capture consumer engagement’s dimensions in a conceptual framework.

 Carman and colleagues, who served as technical assistance providers for the Aligning Forces for Quality (AF4Q) program, described 3 levels at which patients can be engaged: the direct care level, in which individuals become engaged in their own health and healthcare; the level of organizational design and governance for healthcare organizations; and finally, the policy-making level, when consumers are involved with federal, state, and local healthcare policy.

 At each level, the consumer role is viewed on a continuum, from consultation with consumers at the low end to partnership and shared leadership at the high end.

Mittler and colleagues, who were part of the AF4Q program evaluation team, further differentiated engagement activities at the direct-care level into 4 components.

 Two relate to health behaviors: self-management behaviors, which focus on management of chronic disease, and healthy behaviors, which are general health-promoting behaviors like healthy eating and engaging in physical activity. The other 2 relate to managing healthcare: healthcare encounter behaviors relate to effectively communicating with healthcare providers and shopping behaviors include making informed choices to select high-quality (or high-value) healthcare providers and treatments.

This article examines how consumer engagement was implemented in the AF4Q program’s 16 multi-stakeholder coalitions (alliances). Specifically, we examine the approaches to implementing efforts in the 4 key areas of consumer engagement in the AF4Q program, which include examples of consumer engagement at the direct-care and healthcare organization governance levels, but not at the policy-making level. At the direct-care level, we examined efforts to improve (1) self-management and (2) shopping; at the healthcare organizational level, we examined efforts to involve individual consumers in (3) alliance governance and (4) healthcare quality improvement teams. The first 3 areas were required as part of the AF4Q program, while the fourth was voluntarily developed by several alliances. We examined how and to what degree the alliances embraced each of these 4 areas of consumer engagement, the factors related to the level of embrace of work in the areas of consumer engagement, and, for the direct-care consumer engagement efforts, we also examined the extent to which the interventions had population-level effects.

With the AF4Q initiative, the Robert Wood Johnson Foundation (RWJF) was early to embrace the concept of consumer engagement. The AF4Q program was premised on the idea that to improve the quality of patient care, consumers needed to be involved with providers and purchasers. Consumer engagement was seen as one of the key drivers for improving healthcare quality in the United States, along with performance measurement, public reporting, and quality improvement.

Although RWJF viewed consumer engagement as essential to the AF4Q program, what exactly consumer engagement would look like within the initiative was less clear.

 The initial call for proposals required “substantial and credible” consumer representation in the alliance leadership.

 It also emphasized consumers’ use of information to make care decisions. To clarify how that would be operationalized, early in the AF4Q program, RWJF created a consumer engagement learning community in which alliances met, received technical assistance, and shared ideas. As part of the learning community, each alliance was expected to develop plans in 2 areas of direct-care consumer engagement: self-management and consumer use of public reports of provider quality.

 RWJF staff later wrote, “We let a thousand flowers bloom, believing that there was too little evidence about what works in consumer engagement to dictate one approach.”

The AF4Q alliances’ early experience with consumer engagement was challenging. According to early evaluation findings, “Developing a concerted, coherent consumer engagement strategy has taken much more time than originally anticipated in virtually all of the sites.”

 The challenges were attributed to differing levels of enthusiasm for the work within alliances, a lack of existing evidence-based strategies, and difficulty reaching consensus on strategies within alliances.

In response to the challenges, RWJF began reworking its approach to consumer engagement in late 2008. Instead of supporting a “wide breadth of activities,” the Foundation decided to provide alliances with more structure and narrower consumer engagement expectations.

 In March 2009, the Foundation sent the alliances a memo clarifying that the principal goal for consumer engagement was raising consumers’ awareness and use of comparative healthcare performance information.

 Specifically, alliances were tasked with providing consumer friendly public reports of healthcare provider performance information and using a range of strategies to connect consumers to access and use the information to facilitate consumer shopping. The memo also reiterated the Foundation’s desire for consumer involvement in alliance governance. It further explained that the new consumer engagement expectations could be viewed as a “floor of activities” and not the entirety of what alliances could do. Thus, alliances could continue to implement programs in self-management or other areas of consumer engagement as long as they also disseminated consumer-friendly public reports on provider quality and involved consumers in governance.

Soon after the memo’s release, the alliances were required to report specifically on metrics related to the 2 main consumer engagement goals. For shopping, alliances had to report on the number of visits and page views for their online public reports of ambulatory provider quality performance. For integrating consumers into alliance governance, alliances had to document that at least 1 consumer (not a consumer advocate) was represented on the alliance’s leadership team or workgroup. As the grant reporting requirements became more process-oriented, the alliances were asked to describe major activities (eg,“to promote consumers’ use of health and comparative performance information in healthcare decision making” and “engaging individual consumers and consumer advocates in the work of [the] AF4Q [program]”).

We used a qualitative approach to examine the alliances’ experiences implementing programs in the 4 areas of consumer engagement. In addition, for consumer shopping for high-quality healthcare providers, we assessed the consumer friendliness of the alliances’ public reporting websites.

For the direct-care levels of consumer engagement, self-management and shopping, we also examined population-level changes over 4 early years of the AF4Q program (2007-2008 and 2011-2012) using 2 rounds of the AF4Q Consumer Survey. (For New Mexico and Boston, which joined the AF4Q program later, the round 1 survey was conducted in 2010 and the round 2 survey in 2013/2014.) Details of the data, analysis, and results are presented in the online 

. It is notable that a third round of the consumer survey had been part of the evaluation team’s research design, but it was not, in the end, funded. Without a third round, the analysis of population-level effects was limited to approximately the first half of the AF4Q initiative. However, our qualitative results of the AF4Q program’s self-management efforts do not suggest that there would be a greater impact on patient activation later in the program. Similarly, we did not observe a substantial change in consumer friendliness of the AF4Q public reporting websites after 2012, as is detailed in the results section later in this paper.

As part of the AF4Q program evaluation, the evaluation team conducted 1100 qualitative interviews with alliance stakeholders over the program’s almost 10-year history. The qualitative interviews were conducted through a combination of periodic in-person site visits and annual or semi-annual telephone interviews. In each round of interviews, alliance stakeholders were asked about their consumer engagement work, and there were 2 rounds of interviews that largely focused on consumer engagement.

The interviews were audio-recorded, transcribed, and coded in Atlas.ti for the key AF4Q programmatic areas (eg, consumer engagement, quality improvement, and public reporting). For each alliance, data related to consumer engagement were retrieved and reviewed by 2 authors, and a detailed summary of how the alliance approached the 4 areas of consumer engagement (and any additional areas) was developed. This summary included qualitative data on alliance stakeholders’ attitudes and perceptions of the various areas of consumer engagement. The 2 authors then characterized each alliance’s level of embrace for each of the 4 areas of consumer engagement, as: (1) high embrace, in which the alliance made the work a key priority; (2) medium embrace, in which alliances made a concerted effort to develop programming, but in a more limited way; or (3) low embrace, in which alliances did the minimum required or “checked the box” in that area. We also identified themes that covered all 4 areas of consumer engagement related to developing consumer engagement programming within the AF4Q initiative.

Assessment of the Consumer Friendliness of Public Reports

To assess the consumer friendliness of the alliances’ public reporting websites, we examined the diabetes quality metrics for ambulatory care at 3 time points: (1) 2010, summer; (2) 2013, summer; and (3) 2015, fall. Diabetes metrics were selected because they were most consistently publicly reported by alliances.

 We included the 14 alliances that reported ambulatory diabetes metrics in each of the 3 time periods (excluding West Michigan and Western New York, which last reported in 2011 and 2014, respectively). We analyzed Wisconsin’s consumer-oriented public report rather than the alliance’s main reporting website.

We used 3 measures of consumer friendliness related to providing simpler and more evaluable information to consumers, which were derived from the research literature available early in the AF4Q program.

 Specifically, we examined the reading level of the description of the diabetes indicators using the Flesch-Kincaid readability test.

 We evaluated the description of the diabetes indicators for technical language, categorizing them into technical (eg, “A1C test”), mixed (eg, “blood sugar [A1C] less than 7%”), and plain English (eg, “blood sugar control for adults with diabetes”). (A1C indicates glycated hemoglobin.) We also assessed whether performance was displayed using an evaluable approach (eg, stars or word icons) or if bar charts with percentages were used, which require greater cognitive processing.

We examined whether there was change in each consumer friendliness metric across the 14 alliances over the 3 time periods. Using 2015 data, we categorized the alliances based upon how many of the following thresholds they met for consumer friendliness: using language that was assessed at below the ninth-grade reading level, use of plain English (rather than technical or mixed), and use of evaluable icons.

Consumer engagement was a new area to almost all of the AF4Q alliances, and few alliance staff had background or expertise in the area. Alliance staff explained: “It’s not our expertise,” “This is all new stuff that we’re kind of playing with because of the Robert Wood Johnson initiative,” and “I don’t think I know much about it, and I don’t think it’s very easy.” This was largely due to the AF4Q request for proposals’ program description and alliance selection criteria, which disproportionately focused on performance measurement and public reporting.

 Not only did the selection criteria require applicants to demonstrate familiarity with nationally recognized performance measures and have the capacity (or progress toward capacity) to collect performance information, they had to commit to publicly reporting performance information for half of their community’s primary care providers within 3 years. In contrast, the only specific requirement for consumer engagement was to have a leadership team with substantial consumer representation.

Selecting alliances based on their commitment to public reporting meant that the organizations rarely had a consumer engagement background and the expertise, and few had existing consumer constituencies. For example, leaders of 2 alliances shared: “We are a physician-led, really not direct-to-consumer kind of an organization” and “We’re not looking to have a consumer constituency of our own.” Some alliances joined the AF4Q program very enthusiastic about developing programming in consumer engagement. One leader said, “When the grant came out we all said, ‘Oh, we need to do more of that consumer engagement thing, whatever that is.’” Others, however, described coming “kicking and screaming,” saying, “My god, how can we make a dent?”

These contextual factors made the development of consumer engagement within the AF4Q program challenging. The challenges were further exacerbated by having clearly defined public reporting goals from the program’s start without having similarly clear requirements for consumer engagement. One alliance leader described the dynamic: “What I was told, literally, is, ‘You need to bring up a public report within a year, and if you don’t, you’re not in the game anymore.’ So even though they were telling us to do consumer engagement…we knew that if we wanted to stay in, we had to focus on public reporting.” In contrast, alliance directors described the consumer engagement expectations as “unfocused,” “inconsistent,” and “not defined.” One alliance leader summed up the situation saying, “If we looked at what…the RWJF wanted us to achieve [in consumer engagement], it was not really clear. Plus, it was unfamiliar territory for us.”

In the following sections, we describe how alliances implemented each of the 4 key areas of consumer engagement. Specifically, we detail an illustrative alliance’s approach at each level of embrace (high, medium, and low) and the factors related to the level of embrace, and for the direct-care level efforts, we examine the population-level impact of the work.

Two alliances (Humboldt County, California, and south central Pennsylvania [SCPA]) strongly embraced chronic disease self-management programming throughout the AF4Q initiative. A third alliance (Western New York) also initially embraced self-management, but due to a change in strategic direction, it stopped self-management work at the end of 2013. The Humboldt County, California alliance, which was the exemplar in this area, decided early in the AF4Q program that “patient self-management is the main part of consumer engagement.” The alliance launched the Stanford Chronic Disease Self-Management Program in over 25 locations throughout the county.

 In this evidence-based program, patients with chronic conditions met 2.5 hours a week for 6 weeks with lay facilitators to interactively learn about topics such as handling frustration, decision making, and how to evaluate a new treatment. By the end of the AF4Q program, 1047 adults, or 1 out of every 103 adults in Humboldt County, graduated from the program (another 508 participated but did not complete the full program).

 A leader attributed this wide reach to building a “grass roots political campaign” to spread the program, including working closely with primary care physicians who would make referrals to the program like they would for a specialist.

These alliances were primed to develop programming in the self-management area. A Humboldt County leader described previously trying to run the Stanford program, but having “failed”: “You learn from the first failure about what works.” Both the Humboldt County and SCPA alliances had “very influential physician champions” supporting consumer engagement work. Humboldt County also had a volunteer consumer “thought leader” champion and the SCPA alliance hired senior staff with consumer engagement expertise. Both alliances credit the AF4Q initiative with enabling them to develop this programming: “I can’t thank them [RWJF] enough. It would not have happened without that.”

These 2 alliances could have made a population-level impact on patient activation, a term that means having the skills, knowledge, and confidence to manage one’s health and healthcare. We examined the population-level impact using 2 different analytical approaches (

). In the difference-in-differences regression model (

), there was a small and significant improvement in activation in Humboldt County compared with what occurred in the comparison sample (an increase of 4.3 points vs 3.8 points, respectively, on a theoretical 100-point scale). (Fowles and colleagues have found that a 5-point difference in activation is the difference between those who engage in a healthy behavior and those who do not, so the overall increase from round 1 to round 2 is likely meaningful.

) However, in the fixed-effects model, no significant difference was detected (

). Notably, no other alliances had significantly more improvement in activation than the comparison sample.

Most alliances (10) made concerted efforts to develop educational programs in self-management that were typically limited in their reach and/or intensity. The Memphis alliance, for example, published a weekly article in the local newspaper from 2007 to 2014 that focused on health and healthcare-related topics, including managing chronic conditions. For a number of years, the alliance also convened a Neighborhood Outreach Workgroup (NOW) that brought together a handful of community leaders in 4 lower- and middle-income neighborhoods.

“NOW was much more door-to-door, neighbor-to-neighbor, sitting in somebody’s house talking about 10 people at a time—what you need to do about your health.” An alliance leader described the strengths and weaknesses of what they accomplished: “We’ve created a lot of awareness, but in terms of creating activation, there’s still a lot of work.”

Other medium-embrace alliances worked through employers to reach consumers, “partner[ed] with organizations that are more consumer facing than we are,” and used their websites to reach consumers. Among the approaches taken were creating pilot educational projects, conducting patient empowerment training, and e-mailing health-related communications to consumers. The medium-embrace alliances took the work seriously, but it was often aimed at a narrow group of consumers (“scale is an issue”), not always a good organizational fit (“it’s sort of been an add-on more than the core of what we do”), and often not sustained during the entire AF4Q program. In fact, 4 alliances abandoned their self-management educational efforts after RWJF’s 2009 memo clarified its interest in efforts to encourage consumer use of public reports. One leader explained that self-management was new to their organization, and because their self-management work didn’t “seem to be what the Foundation’s interested in…we shifted gears.” This same alliance, however, reported that the organization was fundamentally “transformed” by the consumer engagement work: “If we looked back before our work with Aligning Forces…it was just assumed that you were meeting the needs of patients. But I would say now there’s much more direct conversation around what is it that the patients need, how do we ensure that we understand it...”

Finally, 3 alliances minimally embraced the area of self-management. One alliance made posters about diabetes and distributed them to physician practices and another tried several approaches, including scheduling classes on taking control of your healthcare, but “didn’t have much uptake.” One of these alliances stopped its consumer engagement work altogether in the final phase of the AF4Q program, explaining, “I think what we struggled with, as we did with all of these consumer pieces, [was] where we were doing almost—I would say&mdash;one-shot type of things that were good. But the long-term sustainability of pulling it together under a consumer rubric was what we found challenging.”

Consumer Involvement in Alliance Governance.

 This area was most commonly embraced by AF4Q alliances. Half of the alliances enthusiastically incorporated individual consumers in alliance governance, 4 of which changed their bylaws to require 1 or more consumers on the lead organization’s board. The Boston alliance was an exemplar in this area. In November 2011, in response to AF4Q program expectations and the strong voice of some existing involved consumers, the alliance established a board-level consumer council on par with its existing physician and health plan councils. Further, they amended their bylaws so that 2 consumer council members could serve on the organization’s board. An alliance leader described these accomplishments as the alliance’s most successful element of consumer engagement: “I think that’s huge in setting the stage for [the organization] going into the future, that all of our work [is] not only collaborative, but it absolutely includes the patient’s voice.”

Three alliances moderately embraced this area, making a serious effort to involve consumers in governance, but the concept did not permeate the organization. One alliance, for example, convened a consumer workgroup that met 3 times a year and worked on specific deliverables, such as providing feedback on public reports.

While the alliance had a consumer join the board, it did not happen until quite late in the AF4Q program.

Five alliances minimally embraced inclusion of consumers in governance, doing what was required by the AF4Q program, but not necessarily buying into the concept. One leader described putting a consumer on the leadership team when it was required in 2010: “We’ve got a terrific person. And she’s very dedicated to the program. I’m not sure how much she gains or we gain from it, but we’re happy to have her.” Others described having purchasers or employers serve as their consumers (“they [the consumers] tend to kind of look and feel more like a purchaser in terms of their level of understanding”). Another alliance in this group, however, described gaining an appreciation for the consumer perspective from trying to have a consumer on the board. Although they ultimately decided not to have a consumer serve on the board (related to challenges of empowering consumers and getting them up to speed on the issues), the alliance’s staff interviewed consumers to bring consumer perspectives to board meetings when needed.

Consumer Involvement in Healthcare Providers’ Quality Improvement Teams.

 The 2 alliances that strongly embraced self-management also strongly embraced Patient Partner programs, which integrate consumers into ambulatory care quality improvement teams. Although this area of consumer engagement was not required by the AF4Q program, both alliances, interestingly, credited their efforts in this area as being their biggest consumer engagement success. One leader said, “It was a huge game changer to get the patients into the room during the [practice’s quality improvement] collaborative,” and another explained, “It has made, I think, a philosophical change for our community and our healthcare system.”

The SCPA alliance learned about the Humboldt County alliance’s Patient Partner program in 2010, and the leadership was struck by the concept of including patients in physician practices’ quality improvement efforts. A leader explained, “It was sort of, wow, if we want patient-centered medical home[s], they better be patient-centered, and how could they ever be patient-centered if you’re not having patients being involved in the development of the work and the changes?” In 2011, the SCPA alliance began requiring practices participating in their quality improvement program to bring 2 patient partners onto the quality improvement team. This effort grew from 14 patient partners in 7 primary and specialty care practices in 2011 to 70 partners in 58 practices in June 2016 (more than a year after the AF4Q program ended). The SCPA alliance staff and steering committee trained the patient partners, did monthly webinars or face-to-face meetings to support them, and troubleshooted with practices if problems arose.

Three other alliances also created programs similar to Humboldt County’s and SCPA’s Patient Partner programs. One required patient-centered medical home pilot practices to include 2 or more patient or family members in their redesign efforts, although the alliance did not work directly with the consumer members. Another alliance had a short-term program, and a third started a small effort in the final year of the AF4Q program. The remaining 11 alliances chose not to integrate patients into quality improvement efforts.

Shopping: Consumer Friendliness of Public Reports.

 shows the metrics for consumer friendliness of the alliances’ public reporting websites from 2010 to 2015. There was not substantial improvement in consumer friendliness over the time period. While the number of alliances that used technical language to describe diabetes metrics decreased from 4 to 2, the reading level increased slightly from an average grade level of 7.6 to 8.1 and only 1 alliance switched from using percentages to evaluable icons.

In 2015, only 2 alliances, Maine and Oregon, met all 3 criteria for consumer friendliness (reading level below a ninth-grade level, used plain English to describe quality indicators, and had evaluable icons) (

). Another 5 alliances (Boston, Minnesota, Washington, Kansas City, and Humboldt County) met 2 of the 3 criteria. Of these 7 alliances, 5 were publicly reporting ambulatory care quality or were in the process of doing so when the AF4Q program began. One alliance leader explained that when their existing site fared poorly in RWJF testing with consumers, they were motivated “to totally revamp the website.” Another described the AF4Q initiative as pushing them “to put equal weight into getting information to consumers…so this is huge, and I don’t think it would have happened if it hadn’t been for Aligning Forces.”

The consumer friendliness of these alliances’ websites, however, did not result in significantly larger increases in consumer use of doctor or hospital public reports in these communities compared with the national comparison sample (Greene J, et al; unpublished manuscript). There was, notably, an increase in awareness of physician quality reports in the AF4Q communities (from 13.3% to 16.8%, 3.5 percentage points); however, it was no larger than the gain in the comparison sample (from 11.5% to 17.2%, a 5.7-percentage point gain). No AF4Q community increased consumer use of public reports significantly more than what was observed in the comparison sample over the first half of the AF4Q initiative, the period covered by the consumer survey data.

Six alliances met one of the consumer friendliness criteria and 1 alliance did not meet any of the criteria (2 alliances were not reporting in 2015). Among these alliances with less consumer friendly websites, some leaders described facing barriers from physicians: “So, for example, we know that best practices are not just simply to give data to consumers and have them draw their own conclusions. Well, on our website, we give data to consumers and they have to draw their own conclusions unfortunately…we have not been able to get stakeholder agreement around that [labeling providers as above average, average, or below average].” Other alliances cited the high cost of redesigning their website as a barrier to making it more consumer friendly. One leader said, “I’d like to have us move toward that direction [ie, consumer friendly design], but we didn’t budget for those kinds of updates and changes, so we’re continuing to use what we’ve got.”

Finally, the Wisconsin alliance met only 1 of the consumer friendliness criteria; however, it created a separate consumer-oriented website with cartoon stories about the importance of quality measures and taking charge of one’s healthcare.

 This alliance, which was publicly reporting quality prior to the AF4Q program, decided it “needed to do something different that really met people where they were at.” Despite developing a consumer-focused website that was generally very accessible to consumers, the alliance utilized some display approaches, like bar charts, that have been shown to be challenging for consumers to process, particularly those with low numeracy skills.

Consumer engagement in the AF4Q program was challenging because few alliances had consumer engagement expertise or existing consumer constituencies, the Foundation’s consumer engagement expectations were not always specific, and there was a limited evidence base on effective interventions. Despite these obstacles, there were a number of successes in the AF4Q program’s consumer engagement efforts. Perhaps most notable is that the majority of the alliances described the AF4Q initiative as sparking a cultural shift within their organizations, in which the consumer perspective became more important and integrated into many aspects of the alliances’ work. Instead of assuming that the alliance was meeting the needs of patients, there was a realization that it was important to seek out and learn from individual consumers about their point of view. This occurred within alliances that strongly embraced multiple elements of consumer engagement, but it also seemed to appear within alliances that did not feel it was their role to run self-management programs or have consumers on their organization’s board. This shift has positioned the alliances to integrate consumer perspectives into their future work.

Another key success was sparking 2 alliances, Humboldt County and SCPA, to embrace self-management programming and develop Patient Partner programs that incorporated consumers into quality improvement efforts in ambulatory care settings. The AF4Q program’s support enabled these alliances to develop robust consumer engagement efforts, learn from each other, and inspire other AF4Q alliances and other organizations nationally. These alliances were primed to develop consumer engagement programming and had strong leadership support and prior expertise or attempts at consumer engagement. The resources from the AF4Q program enabled them to dramatically grow their programming and commitment to consumers. In Humboldt County, the alliance’s efforts may have made a population-level impact: because the county has a relatively small population, the 6-session self-management program was completed by 1 of every 103 adults.

The AF4Q initiative prompted some success in creating consumer friendly public reports of provider quality performance. Five of the 6 alliances that were reporting ambulatory care quality or en route to doing so prior to the AF4Q program created websites that adhered to consumer friendly design principles, and the sixth alliance created an innovative cartoon guide helping consumers understand the importance of quality measures. Alliances new to public reporting described being very focused on the challenges of simply producing a public report, but not having the bandwidth, resources, or stakeholder commitment (and, in some cases, interest) to make it consumer friendly. Unfortunately, however, the consumer friendliness of the websites did not translate to increased consumer use of comparative quality information—at least over the time period covered by our data. Many alliance leaders highlighted that public reporting was not like the quote from the movie 

, where “If you build it, [they] will come.” Few alliances had consumer constituencies that they could draw to the website, and efforts with 

 in 3 communities suggested that working with established consumer organizations was a more effective way to reach consumers (Greene J, et al; unpublished manuscript).

From Rhetoric to Reality: Consumer Engagement in 16 Multi-Stakeholder Alliances

 To summarize the results from the quantitative analyses conducted during the summative evaluation of the Aligning Forces for Quality (AF4Q) initiative.

 Longitudinal design using linear difference-in-difference (DD) regression models with fixed effects. Outcomes were selected based on the AF4Q program logic model and organized according to the categories of the Triple Aim: improving population health, improving quality and experience of care, and reducing the cost of care.

Two primary data sources: the AF4Q Consumer Survey and the National Study of Physician Organizations (NSPO); and 4 secondary data sources: the Dartmouth Atlas Medicare claims database, the Truven Health MarketScan commercial claims database, the Behavioral Risk Factor Surveillance System (BRFSS), and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS).

In total, 144 outcomes were analyzed, 27 were associated with improving population health, 87 were associated with improving care quality and experience, and 30 were associated with reducing the cost of care. Based on the estimated DD coefficients, there is no consistent evidence that AF4Q regions, over the life of the program, showed greater improvement in these measures compared with the rest of the United States. For less than 12% of outcomes (17/144), the AF4Q initiative was associated with a significant positive impact (

≤.05), although the magnitude of the impact was often small. Among the remaining outcomes, with some exceptions, similarly improving trends were observed in both AF4Q and non-AF4Q areas over the period of intervention.

Conclusion and Policy and Practice Implications: 

Our quantitative findings, which suggest that the AF4Q initiative had less impact than expected, are potentially due to the numerous other efforts to improve healthcare across the United States, including regions outside the AF4Q program over the same period of time. The limited overall impact may also be due to the variability in the “dose” of the interventions across AF4Q regions. However, these results should not be interpreted as a conclusive statement about the AF4Q initiative. More nuanced discussions of the implementation of interventions in the specific AF4Q programmatic areas and their potential success (or lack thereof) in the participating communities are included in other articles in this supplement.

After almost 10 years, the Robert Wood Johnson Foundation’s (RWJF’s) Aligning Forces for Quality (AF4Q) program, the largest privately funded community-based healthcare initiative to date, ended in April 2015.

 The AF4Q initiative’s design assumed that efforts to improve health and healthcare would be more effective when key stakeholders in a community collaborate to design, coordinate, and implement various interventions aimed at improving health system processes and outcomes.2 Between July 2006 and April 2010, 17 multi-stakeholder coalitions (alliances) in communities representing 12.5% of the US population were competitively selected as grantees; 16 were still part of the program when the AF4Q initiative ended. Over the life of the AF4Q program, RWJF dedicated $300 million in the form of direct grants to participating alliances; payments for technical assistance, program administration, and communication; and funds for an independent evaluation.

The design and implementation of the AF4Q initiative might be characterized by the program’s complexity and ambitious scope and the participating communities’ diverse geography and distinctive contextual factors, which included small rural areas such as Humboldt County in California; large metropolitan areas such as Boston; and 6 states. As discussed in greater detail in this supplement,

 the AF4Q interventions, to be implemented by regional multi-stakeholder alliances, were targeted at 5 primary programmatic areas: performance measurement and public reporting, consumer engagement, quality improvement, health equity, and health system payment reform. According to RWJF’s goals for the AF4Q program and its theory of change,

 focused and planned alignment among these different intervention areas would enhance the effectiveness of the initiative in improving population health outcomes, creating more patient-centered care, and improving the value and efficiency in the use of healthcare resources.

As described by Scanlon et al in an online-only article

 from this supplement, the complexity, scope, and length of the AF4Q initiative created a unique challenge for designing a comprehensive evaluation. Based on the structure and theory of the program, a logic model was developed to provide conceptual links between the various aspects of the program and potentially affected outcomes. The logic model also informed the selection and measurement of outcomes to be studied.

 The evaluation of the AF4Q initiative employed a mixed-methods approach, and numerous articles were published using quantitative and/or qualitative methods to address specific research questions. Some of these studies focused on a particular programmatic area,

 while others addressed important issues across different programmatic areas.

 In this article, from a summative perspective, we present a unified empirical framework to examine the impact of the AF4Q initiative on a broad set of quantitatively measured outcomes, linked to the 3 important aims in healthcare delivery: improving population health, improving quality and experience of care, and reducing the cost of care.

The Triple Aim was proposed by Berwick et al in 2008,

 2 years after the beginning of the AF4Q initiative; hence, it was not mentioned in the original program design. However, the vision and design of the AF4Q initiative was consistent with the pursuit of the Triple Aim.

 More specifically, a central component of the AF4Q program mission was the establishment of local multi-stakeholder alliances operating as the potential “integrators” in selected communities, which has been suggested as a key precondition to achieving progress in the Triple Aim.

 As the AF4Q program proceeded, the Triple Aim became increasingly recognized by some key participants as the ultimate goal of such an initiative. Equally important, the outcomes presented in this paper were selected at the outset of the evaluation based on the logic model and the targeted intervention areas in the AF4Q program and before the concept of the Triple Aim initially appeared in the literature. Nevertheless, in retrospect, these preselected quantitative outcomes fit nicely in the Triple Aim framework, which is therefore used to organize our discussion of the results.

Although the quantitative outcomes together may provide an informative and objective synopsis of how much overall progress occurred in the 16 communities regarding the 3 aims, the analysis in this paper was not meant to capture important contextual factors and qualitative components of the AF4Q program. A comprehensive assessment of the AF4Q initiative is presented in Scanlon et al

 and qualitative assessments of the specific AF4Q programmatic areas can be found in several other papers in this supplement.

The Quantitative Approach in the Summative Evaluation

The design of the quantitative component of the AF4Q summative evaluation followed 3 principles. First, to avoid data mining, the outcomes and analyses were conceptually driven and determined ex ante based on the program logic model. Therefore, these outcomes reflected the commitment of the investigators to the empirical aspect of the evaluation, a practice that has become increasingly important in large-scale evaluation research.

 Second, all outcomes were tracked longitudinally in the AF4Q regions as well as in other regions included as the comparison sample. Finally, to the extent allowed by the different data sets used in the evaluation, we maintained a consistent difference-in-difference (DD) modeling approach across all outcomes. In total, 144 outcomes were measured and analyzed using 6 different data sources.

Guided by the AF4Q logic model, we selected quantitative outcomes with several key considerations. First, the whole set of outcomes collectively reflected the potential overall impact of the AF4Q initiative in key areas of health and healthcare delivery. Second, the selected outcomes could be measured quantitatively with a reasonable level of validity and reliability. Third, the outcomes were reasonably well connected to the main programmatic areas, with priorities given to the known focal areas of the alliances’ interventional activities (eg, outcomes related to diabetes and chronic illness). Finally, each AF4Q alliance was required to produce and publicly release reports comparing provider quality in the region. The measures adopted in these public quality reports, often considered important indicators for healthcare quality in the participating communities, were used as references in selecting the outcomes to be evaluated. However, we did not use the actual scores of the quality measures publicly reported by the AF4Q alliances in our analysis because the reported measures, patient populations covered, number of providers included, and the starting time of reporting all varied significantly across the AF4Q alliance communities.

Among the 144 selected outcomes, 27 were associated with improving population health, 87 were associated with improving care quality and experience, and 30 were associated with reducing the cost of care. Each outcome was also categorized based on whether it might exhibit impact in the intermediate or long term, according to the logic model, and was linked to the relevant AF4Q programmatic area(s). The list of all selected outcomes is included in the online 

. To understand the potential change in health equity among different racial and ethnic groups, another programmatic area of the AF4Q initiative,

 30 of the 144 outcomes were selected for further analysis (details available upon request).

The Aligning Forces for Quality Initiative: Summative Findings and Lessons Learned From Efforts to Improve Healthcare Quality at the Community Level

eAppendix

Participating Faculty

Reporting Provider Performance: What Can Be Learned From the Experience of Multi-Stakeholder Community Coalitions?

From Rhetoric to Reality: Consumer Engagement in 16 Multi-Stakeholder Alliances

The Longitudinal Impact of Aligning Forces for Quality on Measures of Population Health, Quality and Experience of Care, and Cost of Care

Aligning Forces for Quality Multi-Stakeholder Healthcare Alliances: Do They Have a Sustainable Future

Evaluating a Complex, Multi-Site, Community-Based Program to Improve Healthcare Quality: The Summative Research Design for the Aligning Forces for Quality Initiative

The Aligning Forces for Quality Initiative: Background and Evolution From 2005 to 2015

Letter From Donald M. Berwick, MD, MPP, Guest Editor

Leading Multi-sector Collaboration: Lessons From the Aligning Forces for Quality National Program Office

Summative Evaluation Results and Lessons Learned From the Aligning Forces for Quality Program

Lessons Learned About Advancing Healthcare Equity From the Aligning Forces for Quality Initiative

Improving Care Delivery at the Community Level: An Examination of the AF4Q Legacy

The View From Aligning Forces to a Culture of Health

Healthcare Reform Post AF4Q: A National Network of Regional Collaboratives Continues Healthcare Reform From the Ground Up