Jon B. Christianson, PhD

 Some large employers and healthcare analysts have advocated for retail competition that relies on providers competing on performance metrics to improve care quality. Using publicly available performance measures, we determined whether health systems increased the quality of diabetes care provided by their clinics based on performance relative to competitors.

 Our analysis examined publicly reported performance measures of diabetes care from 2006 to 2013 for clinics in Minnesota health systems.

 We obtained data for 654 clinics, of which 572 publicly reported diabetes care performance. Because some clinics did not report performance, we estimated a Heckman selection model. First, we predicted whether or not clinics reported performance. Second, we estimated the effect of relative performance (a clinic’s performance minus the mean performance of clinics in competing health systems) on clinic performance using the results of the reporting model to control for selection into the sample of reporting clinics.

 Although diabetes care performance improved during our study, health systems did not differentially improve the diabetes care performance of their clinics performing worse than clinics in competing systems. This result indicates divergence between high-performing and low-performing clinics. This result does not appear to be due to risk selection.

 Publicly reporting quality information did not incentivize health systems to increase the performance of their clinics with lower performance than competitors, as would be expected under retail competition. Our results do not support strategies that rely on competition on publicly reported performance measures to improve quality in diabetes care management.

Our results suggest that from 2006 to 2013, health systems in Minnesota did not compete on publicly reported diabetes care measures as envisioned under retail competition.

Health systems did not differentially improve the diabetes care quality of their clinics performing worse than those in competing systems. Low-performing clinics fell further behind high-performing clinics.

A variety of reasons, including a lack of consumer awareness of publicly reported performance measures, may dissuade low-performing health systems from focusing on quality competition.

These results do not support strategies of competition on public performance measures as a means of achieving quality gains in diabetes care management.

Many economists hold the belief, first articulated by Kenneth Arrow, that competitive models have limitations in describing healthcare markets. Supporting this reasoning is imperfect information about outcomes, or quality, tied to specific services.

 Not only do patients typically lack information when selecting providers or treatments, but asymmetric information also applies across providers, who may be unaware of how their quality compares with that of competitors.

 These aspects create difficulties in compensating providers on value and therefore discourage them from competing on quality.

Some large employers and analysts have advocated for increased retail competition to control medical care costs and improve quality of care. As summarized by Galvin and Milstein, “…providing consumers with compelling performance data and increasing their responsibility for the costs of care will slow the increase in health care expenditures and motivate clinicians to improve the quality and efficiency of their care.”

 Providers presumably would attempt to increase their performance relative to competitors to attract new patients, or retain existing ones, and to receive preferential treatment in health plan benefit designs that increase access to patients. Low-performing providers would be encouraged to catch up to high-performing providers, quality variation across providers would decrease, and quality throughout a market would rise.

Attempts to address information asymmetries by increasing the availability of performance information may not always result in a competitive effect.

 Providers may instead devote resources to other strategies effective in increasing revenues and patient flows. They may invest in new service lines or acquire physician practices to improve their bargaining position with payers. Some providers may attempt to improve performance by attracting healthier patients or avoiding patients who may be difficult to treat and contribute to lower performance. Furthermore, if providers believe that consumers do not use publicly available comparative quality information, such as provider report cards, when choosing physicians or hospitals, they might then be less likely to make quality improvement decisions with regard to the performance of competitors.

This study addresses whether health systems increased the quality of their clinics in response to their reported performance relative to competitors. Successful retail competition presumably would narrow the gap between low-performing and high-performing clinics.

The health systems and associated clinics in our study are located in Minnesota, a state dominated by a relatively small number of nonprofit integrated delivery systems.

 Minnesota Community Measurement (MNCM), a voluntary stakeholder collaborative, began an annual public reporting program for diabetes care at the clinic level starting with 2006 performance (reported in 2007). A 2008 Minnesota statute mandated reporting on a standardized set of quality measures,

 which ultimately included the diabetes care metrics reported by MNCM beginning with 2009 performance; however, there is no apparent penalty for not reporting. We identified 654 clinics (from 184 health systems and independent clinics) that offered diabetes care between 2006 and 2013, of which 572 reported their performance in at least 1 year. Diabetes performance measures have been publicly available in Minnesota for longer than in any other geographic area and, in a study involving 14 communities, Minnesota had the second-highest level of awareness of diabetes performance measures in 2012.

We assumed that decisions regarding quality improvement, including whether or not a clinic submits reports, are made by health systems. This assumption is supported by the majority of clinics within a given health system beginning to report in the same year (

]). Nevertheless, individual clinic characteristics other than competition measures may influence performance. Therefore, we took into account the health system’s competitive environment and individual clinic attributes.

Public reporting often is voluntary, and even mandated reporting may not result in 100% compliance. Estimating clinic performance using only clinics that submitted reports may lead to bias because of unobserved factors associated with both reporting and performance. To address this issue, we employed a Heckman selection model. In the first stage, we predicted clinic reporting status, allowing it to depend on prior-year reporting status, competitive environment, clinic characteristics, and the performance year.

In the second stage, we predicted clinic performance using a framework similar to that of Kolstad, which estimated how the performance of surgeons changed after obtaining information about competitors through report cards.

 This framework determines how providers respond to their relative performance—in our case, how much better (or worse) a clinic is compared with competitors&mdash;while controlling for patient volume to capture the response associated with patient demand. For both relative performance and patient volume, we used prior-year measures (ie, lagged) to reflect available information (eg, clinics had 2008 performance data in 2009) and to allow time to react to demand changes. Like the first-stage reporting model, performance varies by competitive attributes, clinic characteristics, and performance year. We used the results of the reporting model to control for selection into the sample of reporting clinics. (

Market segmentation could affect performance and, therefore, our results. Some clinics may attract healthier patients or avoid difficult-to-treat ones to achieve higher performance that then would be attributable to changes in patient population rather than quality improvement. For example, some Medicaid patients are less adherent to medications, which could lead to worse clinical performance.

 To examine market segmentation, we re-estimated the model using patient volume as the dependent variable to determine whether volume differentially changed by clinics’ relative performance. If clinics of either relatively high or low performance differentially avoid difficult-to-treat patients whom they perceive will contribute to lower performance, then the results of this sensitivity analysis likely would find those clinics managing fewer Medicaid patients. If patients are not shifting between relatively high-performing and low-performing clinics, then it is unlikely that market segmentation influences our results.

Our measure of performance is the Optimal Diabetes Care (ODC) score. The ODC score is the percentage of patients (aged 18-75 years) with diabetes who simultaneously achieve 5 treatment goals: (1) glycated hemoglobin (A1C) less than 7% (<8% starting with 2009 performance); (2) blood pressure less than 130/80 mm Hg (<140/80 mm Hg starting with 2010 performance); (3) low-density lipoprotein cholesterol less than 100 mg/dL; (4) daily aspirin use, unless contraindicated (includes only patients with ischemic vascular disease starting with 2009 performance); and (5) documented tobacco-free status.

We constructed indicators of relative performance and competitive environment. First, we calculated the mean ODC score of clinics in competing health systems. Each urban and rural health system’s competitors consisted of all clinics in the other systems within 5 miles and 25 miles, respectively, of any one of the system’s clinics. To measure relative performance, we subtracted each clinic’s competitor ODC measure (ie, mean ODC score of clinics in competing health systems) from its own ODC score. We created quintiles for this measure separately for urban and rural settings. Because we cannot measure relative performance for nonreporting clinics, we used the competitor ODC measure by itself in the reporting model. 

 provides further explanation and descriptive statistics for these measures. We also created measures for the percentage of clinics in competing health systems that submitted reports and the number of competing systems to additionally control for competitive environment.

The MNCM data include the annual number of patients with diabetes by clinic, which we used to control for responses associated with patient demand in the performance model. Beginning in 2009, these data include the number of patients enrolled in Medicare, Minnesota Health Care Programs (MHCP) (ie, Medicaid and other programs for low-income families and individuals), and private insurance, which we used to examine market segmentation.

We constructed explanatory variables for the reporting model from available single-year data sources. These measures are excluded from the performance model, in which we employed clinic fixed effects. We used 2009 American Community Survey 5-year estimates to create indicators of each clinic’s potential patient population. These indicators include the mean age of the population and the proportion of the population on any type of public assistance within 5 miles and 25 miles of each urban and rural clinic, respectively. We used a 2012 licensure data set to determine the number of physicians, percentage of specialists, mean physician age, and percentage of female physicians at each clinic. We determined federally qualified health center (FQHC) status and affiliation with a critical access hospital (CAH) through Web searches.

We estimated separate urban and rural models to account for location-driven differences in competition. In the first stage, we estimated reporting status using a probit regression. Our data suggest that many smaller clinics, and clinics perceived to have difficult-to-treat patient populations, were more likely to have delayed reporting until the mandate. Therefore, we interacted the 2009 performance year indicator with stand-alone clinic status, number of physicians, FQHC status, CAH affiliation, and the proportion of the potential patient population on public assistance. Because the mandate had a large influence on reporting decisions, we present average marginal effects on the probability of reporting over 3 periods: premandate (2007-2008), first mandate year (2009), and post mandate (2010-2013). In the second stage, we employed a fixed-effects model to estimate clinic performance that includes the inverse Mills ratio, obtained from the first-stage estimation. Because the fixed-effects model required 2 observations per clinic, we excluded any clinic that reported in only 1 year, including all clinics that began reporting with 2013 performance, resulting in an estimation sample of 288 urban and 244 rural clinics. We present average marginal effects of each explanatory variable on clinic performance.

Of 654 clinics providing diabetes care, 572 (87.5%) reported at least once between performance years 2006 and 2013. Urban clinics were more likely to be early reporters than rural clinics (

): For 2006, 32.1% of urban clinics and 14.8% of rural clinics reported their performance. The number of clinics reporting increased through performance year 2009, the year reporting became mandated, and then leveled off. By 2013, approximately 80% of clinics reported. Those not reporting were smaller, independent clinics that often had a higher percentage of specialists than reporting clinics (

). Of clinics that never reported, more than half were stand-alone clinics.

Although a large improvement in publicly reported performance occurred between 2008 and 2010 (Figure), previous research using these data found that this increase is mainly attributable to changes in the definitions of measures for A1C, blood pressure, and daily aspirin, which made it easier for clinics to achieve the performance goal.

 Adjusting for these definition changes, performance improved modestly over the study period.

We present average marginal effects of the reporting model in 

). Overall, reporting was highly persistent. Urban clinics whose health systems faced higher-performing competitors were less likely to report than urban clinics with lower-performing competitors. Prior to the mandate, each increase of 1 percentage point in the mean ODC score of clinics in competing health systems was associated with a decrease of 0.74 (95% CI, 0.01-1.47) percentage points in the probability of reporting for urban clinics. Most clinics faced competition that was within 5 percentage points of what would, based on performance, be considered average competition (ie, the mean of the competitor performance measure across the sample), implying that relative to a clinic facing average competition, competitor performance typically affected the probability of reporting by less than 4 percentage points. This effect diminished over the study period, although it remained significant.

 presents the average marginal effects for the clinic performance model. These average effects apply to all clinics regardless of whether or not they reported (see 

effects conditional on reporting). Although, on average, clinics improved over time, responses to competitor performance imply divergence between high-performing and low-performing clinics. Clinics that had performed much better than competitors in the prior year improved their performance in the following year more than clinics that had performed similarly to competitors, on average, by 1.90 (95% CI, 1.35-2.61) percentage points in urban areas and 1.35 (95% CI, 0.61-1.94) percentage points in rural areas. The divergence was greatest in urban areas, where clinics that had performed much better than competitors improved their ODC scores by 2.99 (95% CI, 1.96-4.05) percentage points more than clinics that had performed slightly worse than their competitors, and by 4.06 (95% CI, 2.54-5.96) percentage points more than clinics that had performed much worse than their competitors. These results imply that relatively high-performing clinics were improving faster than low-performing clinics.

We found no significant effect of patient volume on performance, suggesting that clinics did not increase their performance in response to greater or fewer patients in the prior year. The coefficient on the inverse Mills ratio (eAppendix A Table 3) for urban clinics was 0.33 (95% CI, 0.03-0.62), implying that reporting clinics had higher performance than nonreporting clinics. The inverse Mills ratio was not significant for rural clinics.

We only found significant associations between patient volume and relative performance in urban areas (

). Among all payers, clinics that had performed much better than competitors gained, on average, 16.1 (95% CI, 7.2-26.4) patients with diabetes compared with clinics that had performed similar to competitors, and a similar number of patients was gained by clinics performing slightly better than their competitors. Analyzing volume by payer (from 2009 onward), gains in volume were attributable only to privately insured patients. These results imply that high-performing clinics were attracting privately insured patients—a likely intended outcome of public reporting efforts aiming to shift patients to higher-quality clinics. If these patients were relatively healthy, then they potentially contributed to higher performance scores. However, neither relatively high-performing nor low-performing clinics were differentially avoiding MHCP patients, suggesting that the divergence in performance between clinics is not attributable to market segmentation of this more difficult-to-treat population.

We examined whether health systems respond to the performance of their competitors—a behavior expected under retail competition that could lead to quality improvements. Although diabetes care performance improved in Minnesota clinics during our study, clinics that outperformed competitors subsequently improved more than clinics that had performed worse than competitors, indicating a divergence between high-performing and low-performing clinics. This result suggests that public reporting did not incentivize health systems to improve their low-performing clinics in response to competing against high-performing clinics in other systems.

Our results differ from those of Kolstad, who found that surgeons improved their mortality rate if they were performing worse than expected after the introduction of report cards.

 However, differences between surgical mortality and diabetes outcomes likely limit the comparability of these findings. Compared with individual surgeons, health systems and their associated clinics may also have access to a variety of alternatives to increase revenues or patient flows when faced with publicly reporting performance. For example, they may acquire physician practices or invest in new service lines to attract patients, methods that are unlikely to be available to individual physicians.

 Public reporting may encourage some health systems to take their first steps toward improving diabetes care quality, but these systems may lack the resources needed to develop sophisticated strategies focused on retail competition. Smith and colleagues found that several physician groups had little focus on diabetes care performance prior to reporting as part of the Wisconsin Collaborative for Healthcare Quality.

 These physician groups were likely to implement simple quality improvement strategies when they started reporting compared with the multiple intervention strategies of higher-performing physician groups. In our study, many clinics that were independent or from smaller systems did not begin reporting until it was mandated. Clinics that began reporting with the mandate scored 10 to 30 percentage points lower on the ODC score’s individual measures relative to clinics that began reporting earlier.

Initial quality improvement efforts take time to execute, because they may include improvements in health information technology, changes in office procedures, and recruitment of quality improvement champions, among others. In this study, some health systems may have been undertaking these steps without yet realizing large gains in quality improvement. These smaller-system and stand-alone clinics also may lack resources needed to implement specific interventions needed to compete based on quality.

Some health systems may not believe that public reporting ameliorates imperfect information between consumers and providers. A relatively small percentage of patients use public quality measures,

 and the evidence that public reports influence demand is mixed.

 High-performing clinics differentially attracted privately insured patients in our study, although these effects are likely relatively small in terms of total revenue, considering that patients with diabetes represent a fraction of patients. In addition, neither MHCP nor Medicare patients shifted from low-performing to high-performing clinics, therefore reinforcing concerns about the usefulness of public reporting for publicly insured patients, including the Medicare Payment Advisory Commission’s contention regarding quality reporting in the Merit-based Incentive Payment System.

 Health systems may believe that the current level of awareness and engagement is below the threshold needed to make investments in quality competition preferable to other uses of quality improvement resources. This explanation is supported by the evaluation of the Robert Wood Johnson Foundation’s Aligning Forces for Quality (AF4Q) initiative, in which MNCM participated. At the end of between 4 and 8 years of participating in AF4Q, community coalition leaders generally “…did not believe that the ‘competitive market strategy’…would improve provider quality or efficiency. In their experience, too few consumers sought out and used the information in this way…”

 One alternative to attracting patients and improving quality is mergers and acquisitions that allow for horizontal and vertical integration. During this study, larger health systems acquired several smaller systems and stand-alone clinics. These acquisitions were likely mutually beneficial, as they increased the market share of the larger health systems and improved performance at the acquired clinics.

There is potential that competitor performance was mismeasured. Health systems may not view some clinics in their market areas as competitors. For example, a large integrated delivery system may not treat small independent clinics as competitors. However, in our study, a handful of large health systems dominate each market and stand-alone clinics comprise only 13% of clinics.

A complete model of competition ideally would incorporate price and quality information in relation to competitive responses. Although health systems may have attempted to adjust prices to gain bargaining power, MNCM did not report total cost measures until after the conclusion of this study.

 Health systems would have had little reason to adjust quality based on competitor pricing, as it is doubtful that consumer decisions would be based on prices without the appropriate information available.

The substantial presence in this study market of large health systems may raise questions about generalizability to areas where smaller health systems are more dominant. However, recent trends have shown an increase in mergers and acquisitions throughout the United States, making vertically integrated health systems and concentrated markets increasingly the norm.

Unique aspects of the healthcare market make it difficult to reward and incentivize quality improvement as envisioned in the competitive market paradigm.

 Even when market information asymmetries were addressed through public reporting, we find that health systems did not compete on quality as proponents of retail competition intended. Although public reporting may incentivize quality gains in diabetes care management through other mechanisms, relying on it to promote retail competition among physicians on performance measures is unlikely to be an effective strategy.

Division of Health Policy and Management, University of Minnesota School of Public Health (DJC, JBC), Minneapolis, MN; Department of Health Management and Policy, University of Michigan School of Public Health (JSM), Ann Arbor, MI; Children’s Minnesota Research Institute (MDF), Minneapolis, MN.

 This research was funded in part by a grant from Pennsylvania State University as part of the evaluation of the Robert Wood Johnson Foundation’s Aligning Forces for Quality initiative. The funding sources had no role in the design and conduct of the study; collection, analysis, and interpretation of data; or preparation of the manuscript.

Articles

The authors determined whether Minnesota health systems responded to competitors’ publicly reported performance. Low performers fell further behind high performers, suggesting that reporting was not associated with quality competition.

Do Health Systems Respond to the Quality of Their Competitors?

 This analysis assessed the evolution of public reporting of provider performance in Aligning Forces for Quality (AF4Q) alliances, contrasted alliances that stopped reporting with those that plan to continue, and drew insights from alliance public reporting efforts for the national transparency movement.

 Combined with document review, qualitative research methods were used to analyze interview data collected, over a nearly 10-year period, from the 16 participating alliances.

 AF4Q alliances made their greatest contributions to provider transparency in reporting ambulatory quality and patient experience measures. However, reporting ambulatory cost/efficiency/utilization measures was more challenging for alliances. Alliances contributed the least with respect to measures of inpatient performance. Six alliances ceased reporting at the end of the AF4Q program because of their inability to develop stable funding sources and overcome stakeholder skepticism about the value of public reporting. Insights provided by alliance leaders included the need to: focus on provider, rather than consumer, responses to public reports as the most likely avenue for improving quality; address the challenge of funding the reporting infrastructure from the beginning; explore collaborations with other entities to increase public reporting efficiency; and develop a strategy for responding to efforts at the national level to increase the availability of information

The AF4Q initiative demonstrated that a wide variety of voluntary stakeholder coalitions could develop public reports with financial and technical support. However, the contents of these reports varied considerably, reflecting differences in local environments and alliance strategies. The challenges faced by alliances to maintain their reporting efforts were substantial, and not all alliances chose to report. Nevertheless, there are potential roles for alliances going forward in contributing to the national transparency movement.

In this article, we examine the efforts of voluntary stakeholder coalitions (ie, alliances) to measure and publicly report provider performance as part of the Robert Wood Johnson Foundation’s (RWJF’s) Aligning Forces for Quality (AF4Q) initiative. (Details regarding these alliances and the communities they served are outlined in the article by Scanlon et al located in this supplement.

) Public reporting of provider performance was a key component of the overall strategy of the AF4Q initiative to improve the quality of care in alliance communities (see logic model in 

). Transparent quality measures could encourage consumers to choose higher-quality providers, enhance patient interactions with providers, and stimulate providers to undertake quality improvement activities to avoid losing patients and the stigma that might be attached to poor performance on quality measures. Health plans could use these measures to develop benefit designs that distinguish higher-quality from lower-quality providers (possibly rewarding consumers for choosing higher-quality providers) and implement pay-for-performance programs. This view of the possible benefits of public reporting was similar to that expressed by national advocates of greater provider performance transparency.

Consistent with this emphasis, alliances were selected to participate in the AF4Q program in part because they had experience with, or expressed a willingness to implement, public reporting.

 However, our analysis (further detailed below) found substantial variation in the performances of alliances with respect to AF4Q public reporting. Some met the reporting goals and timelines established by the AF4Q program and planned to continue reporting after the program’s conclusion, others were less successful, and some stopped reporting entirely once the AF4Q program funding ended. Overall, in a separate analysis, we found that consumers in alliance communities had access to more provider performance information than available to residents of comparison communities

 and this remained true throughout the AF4Q initiative.

To place the AF4Q program public reporting efforts in context, we begin by discussing national performance measurement and public reporting efforts prior to, during, and at the end of the AF4Q initiative. We summarize expectations for measurement and public reporting from the AF4Q program and how they changed over the course of the initiative. After describing our methods for data collection and analysis, we address 3 questions: (1) Across AF4Q communities, how did public reporting evolve and what factors drove that evolution? (2) At the end of the AF4Q program, what distinguished alliances that continued to report from those that stopped reporting? (3) What insights can be drawn from the alliance public reporting experience to inform national efforts to increase provider performance transparency?

Reporting health plan quality measures began in the early 1990s

 under the auspices of the National Committee for Quality Assurance (NCQA). The federal Health Care Financing Administration (now CMS) and several states (eg, New York, Pennsylvania, and California) began reporting hospital quality measures during this time period, as well. By 2000, large employers were encouraging the reporting of physician performance as part of an overall healthcare reform strategy.

 Many health plans responded by reporting a limited number of provider quality measures to their members, or by “tiering” providers based on quality measures, but this information usually was not available to the general public. Private firms also entered the reporting arena (eg, Healthgrades and WebMD), with consumers typically paying for full access to their provider performance measures.

The National Quality Forum (NQF) was established in 1999 as a public—private partnership to “create a foundation for consistent data reporting and collection.”

 Incorporating measures developed by NCQA or endorsed by NQF subsequently became the “gold standard” for public reporting. In 2001, under the auspices of the Leapfrog Group, large employers began to publish information on hospital patient safety practices, using data voluntarily submitted by some hospitals. The Ambulatory Care Quality Alliance (the American Academy of Family Physicians, American College of Physicians, America’s Health Insurance Plans, and the Agency for Healthcare Research and Quality), established in 2004, produced a “starter set” of physician quality measures, which it pilot-tested in 2006. In the public sector, Hospital Compare was created by CMS in 2002, and in 2005 the first process of care measures were displayed on the Hospital Compare website, with patient experience measures added soon after.

Clearly, by the time the first alliances were selected for the AF4Q program in 2006, there was considerable momentum for reporting provider performance, especially quality measures. Subsequently, several additional public-sector reporting efforts were initiated, beginning with the establishment of the Chartered Value Exchange (CVE) program by the Bush administration in 2008. Under this program, community organizations could apply for CVE designation, which entailed a commitment to publish provider quality information. In return, CVEs were to receive access to Medicare performance measurement results, along with technical assistance through a peer-learning network. Of the 24 organizations that received the CVE designation by 2012, 11 were AF4Q alliances.

Support for public reporting expanded under the Obama administration’s Affordable Care Act (ACA), which required CMS to share Medicare data with “qualified” local entities for use in reporting provider performance.

 In addition, advocacy groups encouraged the reporting of physician quality measures on ACA health insurance exchanges. Meanwhile, Medicare’s Physician Compare effort provided data on individual physician characteristics: in 2015, patient assessments of care and clinical quality measures became available for practices and groups. This effort also gave consumers access to information on whether individual providers or medical groups participated in various quality programs, including the Physician Quality Reporting System, the Million Hearts initiative, and the Electronic Health Records Incentive Program.

 In addition to its focus on physicians, CMS initiated public reporting for skilled nursing facilities and home healthcare.

Alongside these federal efforts, 43 states have developed, or plan to develop, all-payer claims databases, providing a resource that could be used in producing provider performance reports.

 Subsequent to the establishment of the AF4Q program, private-sector purchasers increased their support for the reporting of measures of cost and efficiency, in addition to quality measures,

 with the hope that consumers would consider “value” (defined as quality relative to dollars spent) when making their choices and that providers would be encouraged to compete based on the value of the services they provided.

Two of the first 4 alliances selected to participate in AF4Q were already publicly reporting provider performance, and a third was in the process of measure construction. However, there were still relatively few community coalition public reporting efforts nationally at the onset of the AF4Q program, although some coalitions were actively involved in Leapfrog reporting efforts.

 There were several aspects of locally produced reports that appeared promising (ie, adding value to national reporting efforts) and deserving of support in the AF4Q initiative. Locally produced reports could increase the number of different sources of performance information available in the community, raising the likelihood that consumers would become aware of provider performance measures. It was hoped that the reports would draw significant attention, because they were “locally developed,” and that providers would view them as credible, especially when they were involved in measure selection or development. Finally, to the degree that local reporting efforts were guided by input from community stakeholders, their measures might be more salient to community residents than those in national reports.

When they joined the AF4Q program, alliances were charged with publicly reporting (within 3 years) measures of ambulatory quality in the treatment of chronic conditions for at least 50% of primary care physicians in their communities. Alliance leaders felt that their continued funding could very well depend on complying. Almost all alliances reached this goal, although report content varied considerably.

 Subsequently, expectations for alliances were expanded to include reporting patient experience measures for ambulatory care, hospital quality and patient experience measures, and measures of resource use, charge, price, cost and/or efficiency in inpatient and outpatient settings. Alliances were encouraged to make their reports “consumer -friendly” and to pursue different methods of disseminating reports. By the end of the AF4Q program in 2015, the alliances were expected to produce public reports that facilitated consumer selection of “high-value” providers.

In summary, the reporting goals for AF4Q alliances were very ambitious. The hope was that alliances would play a leading role in the emerging transparency movement by providing “models” for public reporting that could be adopted in other communities. During the nearly 10-year AF4Q initiative, the number of NQF and NCQA performance measures of all types expanded considerably, which aided alliances in selecting and constructing measures for their reports. In addition to funding, the AF4Q program provided alliances with technical assistance to support their reporting efforts.

We used 3 data sources in our analyses. We tracked the contents of alliance reports throughout the AF4Q program and constructed a longitudinal dataset for each alliance that contained information on the types of measures reported, start and stop dates for reporting of measures, level of reporting (eg, medical group or medical practice), data sources used to construct each measure (eg, claims, medical records, surveys), and frequency with which measures were reported (eg, annually, biannually). Using this dataset, we examined how reporting evolved across alliances and over time.

Beginning in 2006, and continuing through 2016, the evaluation team conducted 1100 semi-structured in-person and telephone interviews with alliance leaders, staff, and community stakeholders, with most addressing issues related to public reporting. This included 4 rounds of site visits to AF4Q communities, 10 rounds of telephone interviews with alliance leaders between 2007 and 2014, and 2 rounds of telephone interviews between 2012 and 2014 with alliance leaders specifically charged with managing pubic reporting efforts. The semi-structured interviews provided information about alliance commitment to reporting, alliance reporting strategies, and the challenges alliances encountered. Latter rounds of interviews also addressed alliance decisions about whether or not to sustain reporting after the AF4Q program and provided insights regarding how multi-stakeholder coalitions might contribute most effectively to national provider transparency efforts. Telephone interviews also were conducted with key national thought leaders in healthcare to elicit their views on the impact of alliance reporting efforts. (The process for collecting and analyzing these interview data is described in detail in the summative research design article, which is available online at www.ajmc.com.

Our third source of data consisted of AF4Q program documents, including (1) information on guidelines and performance targets established by RWJF and the AF4Q’s national program office, (2) funding proposals submitted by alliances throughout the course of the AF4Q program, and (3) progress summaries that alliances submitted to the national program office every 4 months beginning in 2011.

Evolution of Public Reporting Across Alliances: Achievements and Challenges 

Of the 16 alliances, 5 reported provider performance (ambulatory or inpatient) prior to entering the AF4Q program, and 2 more were engaged in measure selection and development, with the intent of reporting; the remaining alliances began the public reporting process after joining the AF4Q initiative. Given their different starting points, it is not surprising that strategies for reporting, and the contents of reports, varied across alliances. Reports also evolved over time in response to new AF4Q program expectations, fluctuating stakeholder support, and experience gained with reporting and its consequences.

As in previous work, we view alliance report production as beginning with measure selection and continuing through measure construction and dissemination, with alliances revisiting their decisions during the AF4Q program (

). We extend findings from prior work, which included alliance reporting activities from 2006 to 2012,

 by addressing the latter part of the AF4Q initiative (2012-2015; see tables in 

). In the first section, we focus on measure selection, construction, and dissemination, examining 3 types of measures (quality, patient experience, and cost/efficiency/overutilization) for ambulatory and inpatient care. We discuss the evolution in reporting for each type and the important factors influencing that evolution.

Selection and Construction of Ambulatory Measures 

 Consistent with AF4Q program expectations, by 2012, all alliances reported measures of quality of care for patients with chronic conditions.

 One respondent observed, “That’s what you had to do to get the grant.” But at this stage, the reports varied considerably in their scope and sophistication. For example, one alliance that reported measures prior to the AF4Q program included 24 process and intermediate outcome measures relating to vascular disease, diabetes, hypertension, and kidney disease in its report, while an alliance that joined the AF4Q program later, and had no prior reporting experience, reported 4 diabetes process measures. Overall, most of the alliances increased the number of publicly reported quality measures between 2012 and the end of the program, while 3 alliances reduced the number of measures and 2 stopped reporting altogether.

By 2012, all 16 alliances were using nationally endorsed measures in their reports, and in that year, 6 of the alliances began reporting composite performance measures of varying types, in addition to disaggregate measures. These approaches remained unchanged through 2015. Alliances used claims data and medical records data to construct measures, with no clear trend toward more alliances using one source or the other during the nearly decade-long initiative. In 2015, 8 alliances relied on administrative data, 5 used medical records, and 1 combined these data sources.

These were the data sources used by the same alliances in 2012, as well. In contrast, some alliances altered the level at which they constructed and reported measures. In 2012, 5 alliances reported quality measures only at the medical group or large physician organization level, while 10 reported at the practice level and 1 at the individual level. As of 2015, of the 14 reporting alliances, 13 reported at the practice level and 1 reported only at the physician organization level. (The alliance that reported at the individual physician level stopped reporting altogether.)

Alliance efforts to include information from more patients and providers in measure construction were met with several obstacles. Because supplying data was costly, maintaining participation on the part of data suppliers (health plans or providers) often was challenging for alliances. When claims data were used for measure construction, alliances needed to integrate data provided voluntarily by health plans, Medicaid programs, and/or self-insured providers. Therefore, depending on the decisions of data suppliers, alliance reports at different points of time could reflect care provided to different subsets of community residents. One respondent noted that, by the end of the AF4Q program, health plans in the community had just grown “weary” of participating in alliance public reporting efforts, while another observed that a number of large self-insured employers in its community had declined to supply data for public reporting. Alliances that depended on providers to submit data from medical records for measure construction faced a similar challenge and, based on observations of interview respondents, the percentage of participating providers appeared to vary considerably across alliance communities. (Alliances typically did not track this number.) Consequently, there was variation in the ability of consumers within different alliance communities to compare performance across providers.

 Some alliance leaders felt that reporting ambulatory patient experience measures was a valuable extension of their reporting efforts. These measures were applicable to all consumers, in contrast to the much more limited applicability of quality measures for a specific chronic illness. In 2009, 2 alliances participated in a 

 pilot effort to test the feasibility of reporting patient experience at the individual physician level. Although neither alliance continued reporting beyond the pilot study, in 2010, the AF4Q program provided supplemental pilot funding to 3 other alliances to incorporate Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) questions into the existing survey instruments of a limited number of medical groups in their communities and to publicly report results. Two of the 3 alliances continued reporting after the pilot program. Stakeholder support for reporting patient experience grew throughout the AF4Q program, partly due to a national focus on improving the experience of patients in the healthcare system.

 By 2015, 12 of the 16 alliances had reported ambulatory patient experience measures at least once, and 5 reported on a regular basis. A leader of one of these 5 alliances said that ambulatory patient experience reporting may have been their primary contribution to provider transparency in the community.

The development and increasing use of CG-CAHPS measures at the national level made measure selection less of an obstacle to reporting. Despite this, fielding patient surveys to collect the data needed to construct these measures added to alliance reporting costs, and finding the necessary funds proved to be a significant obstacle for many alliances. Alliances responded by using different combinations of membership dues, grants, and state support. However, one respondent expressed hope for the future, observing that hospital acquisitions of physician practices made funding easier to obtain; hospitals possessed more resources to support the effort than did individual physician practices and had a history of gathering patient experience data.

Alliances were asked to report NQF-endorsed measures of resource use, charges, price and cost, or efficiency by July 2011, and to report for 50% of community providers by 2013. By 2012, 7 alliances had accomplished this objective by reporting measures of overutilization, with 1 also reporting a cost measure. The alliances typically reported 4 or fewer measures, with “avoidance of antibiotic treatment for bronchitis” and “appropriate imaging for low back pain” being the most common. By 2015, 9 alliances were reporting measures in this category, with the number and types of measures changing very little between 2012 and 2015, with 2 exceptions. One alliance added 3 cost measures (total cost of care for all patients, total cost of care for adults, and total cost of care for pediatrics) and a second added total cost of care for adults. Of these 9 alliances, 6 reported at the practice level, 2 at the group level, and 1 at both levels.

In general, alliances found that reporting ambulatory cost/efficiency performance was a daunting task, relative to reporting ambulatory quality measures. There were fewer nationally endorsed measures to draw from, and stakeholder support (especially support from health plans that sometimes viewed cost data as proprietary) was more difficult to achieve. One alliance respondent observed that “utilization and price [are] way closer to the pocketbooks of organizations and individuals than quality.” Constructing cost/efficiency measures was technically challenging, as well. Alliances that used claims data to report ambulatory quality measures employed the same source to construct cost/efficiency/overutilization measures where possible. (However, some faced contractual restrictions on use of claims data from health plans for this purpose.) None of the alliances that used only medical records data to report quality measures issued any reports containing ambulatory cost/efficiency/overutilization measures. Some tried to access state-based all-payer claims databases for this purpose, but faced limitations on who could use these data and for what purposes.

Beyond issues related to data access and the complexity of measure construction, arguably the largest obstacle to reporting in this area was a lack of agreement among alliance leadership and stakeholders concerning the usefulness of the effort. One alliance leader summarized this point of view: “I think the biggest challenge we have is that nobody seems to care. The consumer doesn’t use it, and nobody wants to pay for it.” In contrast, 1 alliance played a leadership role nationally in reporting an ambulatory “total cost of care” measure, while another felt that reporting cost/efficiency energized the alliance and placed it in a pivotal position to engage in healthcare reform in its community.

Selection and Construction of Inpatient Measures

 In 2008, the Foundation instructed the alliances to add inpatient quality measures to their public reports (and, subsequently, patient experience and cost/efficiency measures). Alliances could meet this new inpatient reporting requirement by using measures from Medicare’s Hospital Compare program; simply providing a link to the Hospital Compare website would not fulfill the hospital quality-reporting objective. Ten alliances reported Hospital Compare quality measures only, and an additional 4 alliances supplemented those measures with measures constructed from other data sources. Another alliance selected measures from other national reports and its state hospital association, and 1 chose not to report inpatient quality at all during this time period. Four alliances used the same technical assistance provider to help them populate their reports with Medicare inpatient measures. They reported an identical set of Hospital Compare measures that remained essentially unchanged throughout the AF4Q program. By the end of the initiative, only 9 alliances were still reporting inpatient quality measures.

Given alliance access to measures in Hospital Compare and other reports for inpatient quality reporting, measure selection was relatively straightforward and measure construction was not required (

). The primary challenge was convincing stakeholders that the effort was necessary. Because hospital quality reporting was already being done by Hospital Compare and other entities, stakeholders were skeptical that disseminating the same measures through alliance websites constituted a significant contribution to quality improvement in their communities.

 Fourteen of the 16 alliances reported inpatient experience, and all used Hospital Compare to meet this expectation. However, although these alliance reports added another point of access to inpatient experience measures for consumers, the measures did not expand the information already available. As with ambulatory patient experience measures, there was strong support among alliance stakeholders for reporting hospital patient experience and, as with measures of inpatient quality, importing the measures from Hospital Compare was relatively straightforward.

 Alliances were given considerable latitude regarding which measures would satisfy reporting requirements in this area. As they did with hospital quality, virtually all of the reporting alliances extracted measures from Hospital Compare. By 2015, 7 of the 16 alliances included these measures in their reports, with Medicare procedure costs and readmission rates being the most common. Measures drawn from Hospital Compare were based only on a subset of community residents—Medicare beneficiaries. Also, Medicare procedure costs are of limited usefulness for consumer decision making.

Once the alliances selected and constructed provider performance measures, they faced the challenge of how to effectively disseminate the measures to potential users, including clinicians, insurers, and consumers (

). All alliances posted their performance measures on websites that they maintained. Alliances sought out media coverage of the release of their reports and found it was an effective way to garner attention. This attention, however, was harder to generate on an ongoing basis as report releases became more routine.

Over time, RWJF placed more emphasis on dissemination of public reporting to consumers. In 2009, the Foundation informed alliances that to increase consumer awareness and use of comparative provider information, they should focus on making their websites more consumer friendly and engage in new outreach strategies. To monitor their success, alliances were required to report on the number of visits to their websites and page views related to their public reports.

 However, a separate analysis suggested that there were not substantial improvements in the consumer friendliness of alliance websites over the last 5 years of the AF4Q program. Alliances that were relatively new to reporting suggested that this was because they needed to devote their resources to report production.

 Also, some alliances remained unconvinced that consumers had interest in this type of information, citing provider resistance to the use of more intuitive, consumer-friendly measures of provider performance (eg, star ratings).

Disseminating reports to consumers was a challenge because few alliances had existing consumer constituencies for support. Instead, alliances sought to reach consumers through employer stakeholders, traditional media, and social media. The level of commitment to doing so was mixed, as not all alliances felt that the information in their reports was likely to be valued by consumers. One alliance leader said that healthcare professionals constituted the primary group interested in the reported measures; they were “the real consumer group that we’ve actually targeted.” In contrast, several alliances tried innovative approaches to make their reports more accessible to consumers. These included having a reporting website targeted only at consumers and including educational health messages to complement specific provider measures on the website.

The effort that received the greatest attention was an experimental collaboration between the Consumers Union (the policy and action division of 

) and 3 alliances to include their performance measures in local issues of 

 magazine. This generated a very large number of new visits to the alliance reporting websites, suggesting that there was a potential market for their performance data. Building on the Consumers Union pilot study, RWJF funded the DOCTOR Project, which provided guidance and funding for an additional 8 AF4Q alliances and 2 other stakeholder collaboratives to distribute their provider performance data through 

Summary of the Evolution of Alliance Public Reporting Efforts

AF4Q alliances increased consumer access to measures of ambulatory quality and patient experience; through their reporting and dissemination activities, they expanded both the type and amount of information available to consumers (

). In a separate analysis, we compared public reporting activities in alliance communities with a set of comparison sites. We found that consumers in alliance communities had access to more information overall, with the difference remaining relatively stable over time. The exception was patient experience, where the advantage enjoyed by consumers in alliance communities grew. For more information, see Christianson et al.

To date, alliances have made fewer contributions to ambulatory cost/efficiency/overutilization transparency. A small number of alliance leaders remain enthusiastic, but the challenges seem likely to discourage a majority of alliances from expanding their efforts in this area. Most alliance leaders and stakeholders viewed inpatient performance reporting as an area already “occupied” by other state or national organizations. Consequently, one respondent observed that the alliance “checked the boxes” on reporting inpatient measures so as to not jeopardize continued AF4Q funding; this is consistent with limited stakeholder support for more ambitious efforts. Report dissemination efforts varied widely, but, in general, alliances struggled to develop and implement effective means to reach consumers. Explanations included lack of stakeholder support, skepticism concerning whether consumers valued the information, and competing demands on alliance and stakeholder resources.

Sustaining Public Reporting by Alliances 

It would be reasonable to expect that alliances would sustain their public reporting efforts after the AF4Q program, given the high priority that RWJF assigned to public reporting and the funds and technical assistance the Foundation provided to support alliance reporting activities. At the conclusion of the AF4Q initiative, based on our interview data, 6 alliances seemed highly likely to continue their public reporting efforts, 4 appeared less certain, and 6 were not planning to construct any new reports. In this section, we discuss what distinguishes alliances that plan to continue their public reporting efforts from those that have stopped or plan to stop reporting.

Based on respondent interviews, 2 general themes emerged for most, but not all, of these 6 alliances. First, alliance leadership viewed public reporting as central to the alliance’s identity, or “brand.” Five of the 6 alliances were reporting provider performance, or were in the measure selection process, before they were chosen to participate in the AF4Q program. Their stakeholders were committed to public reporting, independent of AF4Q program expectations and support. One respondent observed that “it’s what we do.” Another person said, “We did it before, and we’ll do it after.” A second theme, consistent with the first, is that most of these alliances had relatively stable funding sources, unrelated to the AF4Q initiative, which could be used for reporting. One respondent characterized the AF4Q program as a “blip” in the alliance’s history and certainly not the only driver of the alliance’s priorities or funder of its activities.

Still, these 6 alliances varied in how they viewed their public reporting futures. In one alliance, leadership felt that the public reporting field in the community was “crowded.” Consequently, the alliance planned to reduce the number of measures reported and the frequency of reporting, focusing mostly on ambulatory patient experience. In another alliance, leaders were working to expand alliance funding from community employers, which they believed to be important for maintaining the alliance reporting infrastructure going forward. In a third alliance, inpatient measures were dropped from its report, as stakeholders felt that they were “least used” among its measures; this alliance was planning to focus future efforts on cost/efficiency/overutilization measurement, where its stakeholders saw greater value. A fourth alliance was playing a significant role in its community’s healthcare reform efforts; its leaders planned to reduce alliance emphasis on public reporting but continue it in the near term. These examples illustrate the significant variation among the alliances that anticipated continuing to report in how they envisioned their reporting futures.

Alliances Where Continued Reporting Appears Less Certain

Four alliances have less certain reporting futures, for various reasons. In one case, the alliance was moving reporting responsibilities to another community entity, raising questions about whether public reporting will be sustained after the transition. Second, in 3 instances where alliance leaders expressed a desire to continue reporting, there was not a clear plan to fund it. All were exploring the feasibility of raising funds from various sources, including health plans, local foundations, providers, and state grants. Two of these alliances lagged behind in updating existing reports and had experienced some erosion of stakeholder participation or funding. The leader of another alliance suggested that it would continue to measure and report privately as part of its quality improvement activities, but expressed skepticism that the public reporting of measures was beneficial to consumers.

Of the 6 alliances that discontinued public reporting, none were engaged in public reporting of provider performance prior to the AF4Q program. In most cases, their leaders and stakeholders initially were skeptical that reporting would lead consumers to choose higher-quality providers. This was true for alliances located in rural areas, where physician practices were full and there was concern about maintaining an adequate physician supply; however, similar sentiments were expressed by respondents associated with alliances in urban settings. Stakeholders saw more promise in devoting community time and resources to other endeavors (such as supporting provider quality improvement activities or identifying and publicizing pressing local healthcare issues), rather than continuing their public reporting efforts.

Reporting Provider Performance: What Can Be Learned From the Experience of Multi-Stakeholder Community Coalitions?

 A key component of the Aligning Forces for Quality (AF4Q) program was engaging consumers in their health and healthcare. We examined the extent to which the alliances embraced 4 areas of consumer engagement: self-management, consumer friendliness of reports of healthcare provider quality, involvement of consumers in alliance governance, and the integration of consumers into quality improvement teams.

We used a largely qualitative approach. The evaluation team conducted 1100 in-depth interviews with alliance stakeholders. Two authors reviewed the consumer engagement data for each alliance to assess its level of embrace in the 4 consumer engagement areas. For consumer friendliness of public reporting websites, we also assessed alliance public reports for reading level, technical language, and evaluable displays. Population-level effects were also examined for self-management and public reporting.

 Consumer engagement was new to most alliances, and few had staff with consumer engagement expertise or existing consumer constituencies. For each area of consumer engagement, some alliances enthusiastically embraced the work, other alliances made a concerted but limited effort to develop programs, and a third group of alliances did the minimum work required. Integrating consumers into governance was the area most often embraced, followed by making public reports consumer friendly. Two alliances strongly embraced both self-management and integrating patients into quality improvement efforts. The AF4Q program did not have greater population level effects from self-management or public reporting than were those observed in a national comparison sample.

 The AF4Q program sparked a few alliances to develop robust consumer engagement programming, while most alliances tried consumer engagement efforts for the first time and developed an appreciation for integrating consumer perspectives into their work.

The concept of consumer engagement has received a lot of attention in recent years, being described as “the blockbuster drug of the century” and a key factor for achieving the Triple Aim.

 The idea that consumers can help improve the efficiency of healthcare delivery by being more involved in their health and making more informed choices about treatments and providers has widespread, bipartisan appeal.

 However, what exactly is meant by the term has been questioned by many, including Munro who wrote: “How we define it will determine whether it’s truly a miracle drug—or just another variant of age-old snake oil.”

Because of the ambiguity in definition, 2 recent papers have sought to capture consumer engagement’s dimensions in a conceptual framework.

 Carman and colleagues, who served as technical assistance providers for the Aligning Forces for Quality (AF4Q) program, described 3 levels at which patients can be engaged: the direct care level, in which individuals become engaged in their own health and healthcare; the level of organizational design and governance for healthcare organizations; and finally, the policy-making level, when consumers are involved with federal, state, and local healthcare policy.

 At each level, the consumer role is viewed on a continuum, from consultation with consumers at the low end to partnership and shared leadership at the high end.

Mittler and colleagues, who were part of the AF4Q program evaluation team, further differentiated engagement activities at the direct-care level into 4 components.

 Two relate to health behaviors: self-management behaviors, which focus on management of chronic disease, and healthy behaviors, which are general health-promoting behaviors like healthy eating and engaging in physical activity. The other 2 relate to managing healthcare: healthcare encounter behaviors relate to effectively communicating with healthcare providers and shopping behaviors include making informed choices to select high-quality (or high-value) healthcare providers and treatments.

This article examines how consumer engagement was implemented in the AF4Q program’s 16 multi-stakeholder coalitions (alliances). Specifically, we examine the approaches to implementing efforts in the 4 key areas of consumer engagement in the AF4Q program, which include examples of consumer engagement at the direct-care and healthcare organization governance levels, but not at the policy-making level. At the direct-care level, we examined efforts to improve (1) self-management and (2) shopping; at the healthcare organizational level, we examined efforts to involve individual consumers in (3) alliance governance and (4) healthcare quality improvement teams. The first 3 areas were required as part of the AF4Q program, while the fourth was voluntarily developed by several alliances. We examined how and to what degree the alliances embraced each of these 4 areas of consumer engagement, the factors related to the level of embrace of work in the areas of consumer engagement, and, for the direct-care consumer engagement efforts, we also examined the extent to which the interventions had population-level effects.

With the AF4Q initiative, the Robert Wood Johnson Foundation (RWJF) was early to embrace the concept of consumer engagement. The AF4Q program was premised on the idea that to improve the quality of patient care, consumers needed to be involved with providers and purchasers. Consumer engagement was seen as one of the key drivers for improving healthcare quality in the United States, along with performance measurement, public reporting, and quality improvement.

Although RWJF viewed consumer engagement as essential to the AF4Q program, what exactly consumer engagement would look like within the initiative was less clear.

 The initial call for proposals required “substantial and credible” consumer representation in the alliance leadership.

 It also emphasized consumers’ use of information to make care decisions. To clarify how that would be operationalized, early in the AF4Q program, RWJF created a consumer engagement learning community in which alliances met, received technical assistance, and shared ideas. As part of the learning community, each alliance was expected to develop plans in 2 areas of direct-care consumer engagement: self-management and consumer use of public reports of provider quality.

 RWJF staff later wrote, “We let a thousand flowers bloom, believing that there was too little evidence about what works in consumer engagement to dictate one approach.”

The AF4Q alliances’ early experience with consumer engagement was challenging. According to early evaluation findings, “Developing a concerted, coherent consumer engagement strategy has taken much more time than originally anticipated in virtually all of the sites.”

 The challenges were attributed to differing levels of enthusiasm for the work within alliances, a lack of existing evidence-based strategies, and difficulty reaching consensus on strategies within alliances.

In response to the challenges, RWJF began reworking its approach to consumer engagement in late 2008. Instead of supporting a “wide breadth of activities,” the Foundation decided to provide alliances with more structure and narrower consumer engagement expectations.

 In March 2009, the Foundation sent the alliances a memo clarifying that the principal goal for consumer engagement was raising consumers’ awareness and use of comparative healthcare performance information.

 Specifically, alliances were tasked with providing consumer friendly public reports of healthcare provider performance information and using a range of strategies to connect consumers to access and use the information to facilitate consumer shopping. The memo also reiterated the Foundation’s desire for consumer involvement in alliance governance. It further explained that the new consumer engagement expectations could be viewed as a “floor of activities” and not the entirety of what alliances could do. Thus, alliances could continue to implement programs in self-management or other areas of consumer engagement as long as they also disseminated consumer-friendly public reports on provider quality and involved consumers in governance.

Soon after the memo’s release, the alliances were required to report specifically on metrics related to the 2 main consumer engagement goals. For shopping, alliances had to report on the number of visits and page views for their online public reports of ambulatory provider quality performance. For integrating consumers into alliance governance, alliances had to document that at least 1 consumer (not a consumer advocate) was represented on the alliance’s leadership team or workgroup. As the grant reporting requirements became more process-oriented, the alliances were asked to describe major activities (eg,“to promote consumers’ use of health and comparative performance information in healthcare decision making” and “engaging individual consumers and consumer advocates in the work of [the] AF4Q [program]”).

We used a qualitative approach to examine the alliances’ experiences implementing programs in the 4 areas of consumer engagement. In addition, for consumer shopping for high-quality healthcare providers, we assessed the consumer friendliness of the alliances’ public reporting websites.

For the direct-care levels of consumer engagement, self-management and shopping, we also examined population-level changes over 4 early years of the AF4Q program (2007-2008 and 2011-2012) using 2 rounds of the AF4Q Consumer Survey. (For New Mexico and Boston, which joined the AF4Q program later, the round 1 survey was conducted in 2010 and the round 2 survey in 2013/2014.) Details of the data, analysis, and results are presented in the online 

. It is notable that a third round of the consumer survey had been part of the evaluation team’s research design, but it was not, in the end, funded. Without a third round, the analysis of population-level effects was limited to approximately the first half of the AF4Q initiative. However, our qualitative results of the AF4Q program’s self-management efforts do not suggest that there would be a greater impact on patient activation later in the program. Similarly, we did not observe a substantial change in consumer friendliness of the AF4Q public reporting websites after 2012, as is detailed in the results section later in this paper.

As part of the AF4Q program evaluation, the evaluation team conducted 1100 qualitative interviews with alliance stakeholders over the program’s almost 10-year history. The qualitative interviews were conducted through a combination of periodic in-person site visits and annual or semi-annual telephone interviews. In each round of interviews, alliance stakeholders were asked about their consumer engagement work, and there were 2 rounds of interviews that largely focused on consumer engagement.

The interviews were audio-recorded, transcribed, and coded in Atlas.ti for the key AF4Q programmatic areas (eg, consumer engagement, quality improvement, and public reporting). For each alliance, data related to consumer engagement were retrieved and reviewed by 2 authors, and a detailed summary of how the alliance approached the 4 areas of consumer engagement (and any additional areas) was developed. This summary included qualitative data on alliance stakeholders’ attitudes and perceptions of the various areas of consumer engagement. The 2 authors then characterized each alliance’s level of embrace for each of the 4 areas of consumer engagement, as: (1) high embrace, in which the alliance made the work a key priority; (2) medium embrace, in which alliances made a concerted effort to develop programming, but in a more limited way; or (3) low embrace, in which alliances did the minimum required or “checked the box” in that area. We also identified themes that covered all 4 areas of consumer engagement related to developing consumer engagement programming within the AF4Q initiative.

Assessment of the Consumer Friendliness of Public Reports

To assess the consumer friendliness of the alliances’ public reporting websites, we examined the diabetes quality metrics for ambulatory care at 3 time points: (1) 2010, summer; (2) 2013, summer; and (3) 2015, fall. Diabetes metrics were selected because they were most consistently publicly reported by alliances.

 We included the 14 alliances that reported ambulatory diabetes metrics in each of the 3 time periods (excluding West Michigan and Western New York, which last reported in 2011 and 2014, respectively). We analyzed Wisconsin’s consumer-oriented public report rather than the alliance’s main reporting website.

We used 3 measures of consumer friendliness related to providing simpler and more evaluable information to consumers, which were derived from the research literature available early in the AF4Q program.

 Specifically, we examined the reading level of the description of the diabetes indicators using the Flesch-Kincaid readability test.

 We evaluated the description of the diabetes indicators for technical language, categorizing them into technical (eg, “A1C test”), mixed (eg, “blood sugar [A1C] less than 7%”), and plain English (eg, “blood sugar control for adults with diabetes”). (A1C indicates glycated hemoglobin.) We also assessed whether performance was displayed using an evaluable approach (eg, stars or word icons) or if bar charts with percentages were used, which require greater cognitive processing.

We examined whether there was change in each consumer friendliness metric across the 14 alliances over the 3 time periods. Using 2015 data, we categorized the alliances based upon how many of the following thresholds they met for consumer friendliness: using language that was assessed at below the ninth-grade reading level, use of plain English (rather than technical or mixed), and use of evaluable icons.

Consumer engagement was a new area to almost all of the AF4Q alliances, and few alliance staff had background or expertise in the area. Alliance staff explained: “It’s not our expertise,” “This is all new stuff that we’re kind of playing with because of the Robert Wood Johnson initiative,” and “I don’t think I know much about it, and I don’t think it’s very easy.” This was largely due to the AF4Q request for proposals’ program description and alliance selection criteria, which disproportionately focused on performance measurement and public reporting.

 Not only did the selection criteria require applicants to demonstrate familiarity with nationally recognized performance measures and have the capacity (or progress toward capacity) to collect performance information, they had to commit to publicly reporting performance information for half of their community’s primary care providers within 3 years. In contrast, the only specific requirement for consumer engagement was to have a leadership team with substantial consumer representation.

Selecting alliances based on their commitment to public reporting meant that the organizations rarely had a consumer engagement background and the expertise, and few had existing consumer constituencies. For example, leaders of 2 alliances shared: “We are a physician-led, really not direct-to-consumer kind of an organization” and “We’re not looking to have a consumer constituency of our own.” Some alliances joined the AF4Q program very enthusiastic about developing programming in consumer engagement. One leader said, “When the grant came out we all said, ‘Oh, we need to do more of that consumer engagement thing, whatever that is.’” Others, however, described coming “kicking and screaming,” saying, “My god, how can we make a dent?”

These contextual factors made the development of consumer engagement within the AF4Q program challenging. The challenges were further exacerbated by having clearly defined public reporting goals from the program’s start without having similarly clear requirements for consumer engagement. One alliance leader described the dynamic: “What I was told, literally, is, ‘You need to bring up a public report within a year, and if you don’t, you’re not in the game anymore.’ So even though they were telling us to do consumer engagement…we knew that if we wanted to stay in, we had to focus on public reporting.” In contrast, alliance directors described the consumer engagement expectations as “unfocused,” “inconsistent,” and “not defined.” One alliance leader summed up the situation saying, “If we looked at what…the RWJF wanted us to achieve [in consumer engagement], it was not really clear. Plus, it was unfamiliar territory for us.”

In the following sections, we describe how alliances implemented each of the 4 key areas of consumer engagement. Specifically, we detail an illustrative alliance’s approach at each level of embrace (high, medium, and low) and the factors related to the level of embrace, and for the direct-care level efforts, we examine the population-level impact of the work.

Two alliances (Humboldt County, California, and south central Pennsylvania [SCPA]) strongly embraced chronic disease self-management programming throughout the AF4Q initiative. A third alliance (Western New York) also initially embraced self-management, but due to a change in strategic direction, it stopped self-management work at the end of 2013. The Humboldt County, California alliance, which was the exemplar in this area, decided early in the AF4Q program that “patient self-management is the main part of consumer engagement.” The alliance launched the Stanford Chronic Disease Self-Management Program in over 25 locations throughout the county.

 In this evidence-based program, patients with chronic conditions met 2.5 hours a week for 6 weeks with lay facilitators to interactively learn about topics such as handling frustration, decision making, and how to evaluate a new treatment. By the end of the AF4Q program, 1047 adults, or 1 out of every 103 adults in Humboldt County, graduated from the program (another 508 participated but did not complete the full program).

 A leader attributed this wide reach to building a “grass roots political campaign” to spread the program, including working closely with primary care physicians who would make referrals to the program like they would for a specialist.

These alliances were primed to develop programming in the self-management area. A Humboldt County leader described previously trying to run the Stanford program, but having “failed”: “You learn from the first failure about what works.” Both the Humboldt County and SCPA alliances had “very influential physician champions” supporting consumer engagement work. Humboldt County also had a volunteer consumer “thought leader” champion and the SCPA alliance hired senior staff with consumer engagement expertise. Both alliances credit the AF4Q initiative with enabling them to develop this programming: “I can’t thank them [RWJF] enough. It would not have happened without that.”

These 2 alliances could have made a population-level impact on patient activation, a term that means having the skills, knowledge, and confidence to manage one’s health and healthcare. We examined the population-level impact using 2 different analytical approaches (

). In the difference-in-differences regression model (

), there was a small and significant improvement in activation in Humboldt County compared with what occurred in the comparison sample (an increase of 4.3 points vs 3.8 points, respectively, on a theoretical 100-point scale). (Fowles and colleagues have found that a 5-point difference in activation is the difference between those who engage in a healthy behavior and those who do not, so the overall increase from round 1 to round 2 is likely meaningful.

) However, in the fixed-effects model, no significant difference was detected (

). Notably, no other alliances had significantly more improvement in activation than the comparison sample.

Most alliances (10) made concerted efforts to develop educational programs in self-management that were typically limited in their reach and/or intensity. The Memphis alliance, for example, published a weekly article in the local newspaper from 2007 to 2014 that focused on health and healthcare-related topics, including managing chronic conditions. For a number of years, the alliance also convened a Neighborhood Outreach Workgroup (NOW) that brought together a handful of community leaders in 4 lower- and middle-income neighborhoods.

“NOW was much more door-to-door, neighbor-to-neighbor, sitting in somebody’s house talking about 10 people at a time—what you need to do about your health.” An alliance leader described the strengths and weaknesses of what they accomplished: “We’ve created a lot of awareness, but in terms of creating activation, there’s still a lot of work.”

Other medium-embrace alliances worked through employers to reach consumers, “partner[ed] with organizations that are more consumer facing than we are,” and used their websites to reach consumers. Among the approaches taken were creating pilot educational projects, conducting patient empowerment training, and e-mailing health-related communications to consumers. The medium-embrace alliances took the work seriously, but it was often aimed at a narrow group of consumers (“scale is an issue”), not always a good organizational fit (“it’s sort of been an add-on more than the core of what we do”), and often not sustained during the entire AF4Q program. In fact, 4 alliances abandoned their self-management educational efforts after RWJF’s 2009 memo clarified its interest in efforts to encourage consumer use of public reports. One leader explained that self-management was new to their organization, and because their self-management work didn’t “seem to be what the Foundation’s interested in…we shifted gears.” This same alliance, however, reported that the organization was fundamentally “transformed” by the consumer engagement work: “If we looked back before our work with Aligning Forces…it was just assumed that you were meeting the needs of patients. But I would say now there’s much more direct conversation around what is it that the patients need, how do we ensure that we understand it...”

Finally, 3 alliances minimally embraced the area of self-management. One alliance made posters about diabetes and distributed them to physician practices and another tried several approaches, including scheduling classes on taking control of your healthcare, but “didn’t have much uptake.” One of these alliances stopped its consumer engagement work altogether in the final phase of the AF4Q program, explaining, “I think what we struggled with, as we did with all of these consumer pieces, [was] where we were doing almost—I would say&mdash;one-shot type of things that were good. But the long-term sustainability of pulling it together under a consumer rubric was what we found challenging.”

Consumer Involvement in Alliance Governance.

 This area was most commonly embraced by AF4Q alliances. Half of the alliances enthusiastically incorporated individual consumers in alliance governance, 4 of which changed their bylaws to require 1 or more consumers on the lead organization’s board. The Boston alliance was an exemplar in this area. In November 2011, in response to AF4Q program expectations and the strong voice of some existing involved consumers, the alliance established a board-level consumer council on par with its existing physician and health plan councils. Further, they amended their bylaws so that 2 consumer council members could serve on the organization’s board. An alliance leader described these accomplishments as the alliance’s most successful element of consumer engagement: “I think that’s huge in setting the stage for [the organization] going into the future, that all of our work [is] not only collaborative, but it absolutely includes the patient’s voice.”

Jon B. Christianson, PhD

Articles

Do Health Systems Respond to the Quality of Their Competitors?

Reporting Provider Performance: What Can Be Learned From the Experience of Multi-Stakeholder Community Coalitions?

From Rhetoric to Reality: Consumer Engagement in 16 Multi-Stakeholder Alliances

The Longitudinal Impact of Aligning Forces for Quality on Measures of Population Health, Quality and Experience of Care, and Cost of Care

Evaluating a Complex, Multi-Site, Community-Based Program to Improve Healthcare Quality: The Summative Research Design for the Aligning Forces for Quality Initiative

The Aligning Forces for Quality Initiative: Background and Evolution From 2005 to 2015

Summative Evaluation Results and Lessons Learned From the Aligning Forces for Quality Program

Are Healthcare Quality "Report Cards" Reaching Consumers? Awareness in the Chronically Ill Population

The Aligning Forces for Quality Initiative: Background and Evolution From 2005 to 2012

Midterm Observations and Recommendations From the Evaluation of the AF4Q Initiative