Published Online: September 22, 2012
Jon B. Christianson, PhD; Karen M. Volmar, JD, MPH; Bethany W. Shaw, MHA; and Dennis P. Scanlon, PhD
To describe the approaches used by the Aligning Forces for Quality (AF4Q) alliances in producing community-based reports of physician quality and to assess the contribution of these reports to existing physician performance information.
The study included semi-structured interviews with alliance stakeholders and tracking of the number and content of physician performance reports in 14 AF4Q initiative communities and 7 comparison communities.
The study used qualitative analysis of interview data and systematic tracking of the number and content of physician performance reports over time.
Report production occurred in several stages including initiation, measure selection/specification, measure construction, and dissemination. The measure selection/specification process was often the first major act undertaken by alliances under the AF4Q initiative grant. Alliances utilized nationally endorsed performance measures and made a strategic decision to gain buy-in with physicians. Alliances have experienced greater difficulty in producing buy-in for patient experience measures. The primary decision point for measure construction was the use of administrative claims data or physician-provided medical records data. Overall, AF4Q alliances have contributed to an increase in physician performance information in their communities.
Our findings suggest that the AF4Q initiative has accelerated the development and content of physician performance measures in AF4Q communities.
(Am J Manag Care. 2012;18:S133-S140)
The public reporting of physician performance has been supported by the Bush and Obama administrations as an important element of healthcare reform.1,2 Since 2006, the Robert Wood Johnson Foundation (RWJF) has encouraged public reporting efforts by community healthcare coalitions (ie, alliances, the multi-stakeholder partnerships receiving funding through the Foundation’s Aligning Forces for Quality [AF4Q] initiative) chosen to participate in its AF4Q initiative3 as one way to address deficiencies in quality of care.4 The value of locally produced versus national-level physician performance reports is based on several premises: (1) local efforts are more effective in reporting physician performance measures salient to the community and add to the information available to community residents when selecting providers; (2) physicians view the results as more credible because they have played a role in report development, leading them to engage in more effective, targeted quality improvement efforts; and (3) locally developed reports receive more local media attention, enhancing visibility and credibility with consumers and increasing the likelihood that they will use the information in making healthcare decisions.
Nevertheless, there is reason to be skeptical that coalitions of diverse stakeholders can produce credible physician performance reports, as previous local, voluntary collaborative efforts to implement health system change strategies have had mixed success.5-7 Also, because data submission by providers or health plans is voluntary, the process of producing credible comparative performance reports is fraught with political and technical challenges for even well-funded, historically effective community healthcare coalitions.
With the exception of recent work by Young,8 relatively little has been written about the community-based reporting process and whether resulting reports add significantly to the amount and relevance of physician performance information available to consumers. This article contrasts the approaches that different AF4Q alliances have taken in producing community-based reports with clinical quality and patient experience measures, and assesses the contribution of these reports to existing physician performance information.
The AF4Q initiative provides guidance, technical assistance, and funding to community coalitions, in the hope that this will accelerate their provision of physician performance information, which will change consumer and provider behaviors, thereby improving healthcare quality (see eAppendix A at www.ajmc.com, AF4Q Initiative: Public Reporting Logic Model). When the AF4Q initiative began, only 3 of the initial 14 AF4Q alliances were publicly reporting ambulatory quality data. Nationally, there were relatively few local or state-level efforts to measure and report physician performance. Upon joining the AF4Q initiative, alliances were given a goal to publicly report ambulatory quality measures for over 50% of primary care providers in their communities within 3 years; it was implied that funding could be discontinued if this target was not met.9 The AF4Q initiative provided technical assistance, including webinars on the selection and construction of ambulatory quality of care measures. Later, the AF4Q initiative required the alliances to add patient experience measures to their physician performance data, along with expanding the scope of the reports to include hospital quality and provider efficiency measures. But, due to the urgency attached to the early reporting of physician performance data, and the fact that alliances had more experience in this area than in hospital reporting, this article focuses only on reporting of physician performance measures by the 14 original alliances (data from additional alliances that joined in 2009-2010 are not included because of their limited experience as AF4Q participants).
Some alliance leaders believed that the AF4Q initiative placed disproportionate emphasis on achieving its early public reporting target relative to the attention given to the program’s other core areas—quality improvement and consumer engagement—and that its reporting timeline was overly ambitious. Representing this viewpoint, one alliance leader said, “…the real push is this public reporting piece which is the endgame for AF4Q.” Additionally, some alliance leaders did not share the AF4Q initiative’s view of the potential value of physician performance reports. Their skepticism was expressed in comments such as: “You want to engage consumers, but…with the quality data, it’s interesting and it’s sexy but ‘what the hell are they [consumers] supposed to do with it?’”; and, “…I think the original belief was that all you need to do is report, and things will magically get better …” On the other hand, many alliance leaders felt that the AF4Q initiative’s ambitious reporting target helped move stakeholders from general support for the alliance and its mission to specific actions. In the end, all but 1 alliance was successful in disseminating a public report by the 3-year target date covering at least 50% of primary care physicians in the community (Figure); the reporting efforts of the sole unsuccessful alliance were delayed by state-level legal issues related to uses of health plan data. The sections that follow assess how alliances produced their reports and the contributions of the reports to the existing physician performance information in their communities.
The analyses are based on 2 data sources: semi-structured in-person and telephone interviews that provided information about various aspects of physician performance reporting, such as goals, strategies, and processes for measure construction,10 and the ongoing tracking of contents of alliance reports and the reports produced by other organizations in alliance communities. Interview responses were transcribed from audio recordings, and text files were read and coded. Codes related to performance measurement and public reporting included topics such as challenges/barriers to measurement and reporting; clinical quality and patient experience measurement; data aggregation; and data collection. The coded text was entered into Atlas.ti, a software package for qualitative analysis. These data were used primarily in assessing the processes used by AF4Q alliances to construct reports of physician performance.
Data were collected on the presence and content of public reports in areas served by the 14 original alliances. Without knowledge of their public reporting history, 7 areas that were similar to AF4Q communities in location, population size, and demographics also were selected. Each year, beginning in 2007, we reviewed the websites of hospitals and medical associations, healthcare coalitions, quality improvement organizations, state departments of health, and the AF4Q alliances to document public reporting activities. In addition, websites for the 5 largest commercial health plans, which included national plans operating in the AF4Q communities, were examined. In communities where there were fewer than 5 significant plans, websites for plans with membership that together constituted approximately 75% or more of the total private sector health plan enrollment in the area were reviewed. In all (AF4Q and other) communities, we collected information from organizations that sponsor public reports to verify the search findings and gather further details regarding measure sources and construction. We used this information to determine if alliance public reporting efforts contributed to the type and amount of physician performance information available to consumers in AF4Q communities, how the availability of this information compared with other communities, and if alliance reporting changed over time in ways consistent with AF4Q initiative expectations.
Producing Public Reports
Alliance report production occurred in several stages, including initiation, measure selection, measure construction, and dissemination (see eAppendix B, Alliance Public Reporting Process). In this article, we examine the AF4Q initiative efforts in selecting and constructing physician performance measures (dissemination activities are addressed by Mittler et al in this supplement11). For convenience, we discuss these 2 stages separately, but they overlap at points. For instance, while alliances chose measures in areas where care deficiencies have been documented by national studies, the selection of specific measures was guided by early judgments about what types of data were likely to be available for measure construction. In the Table, we summarize the physician performance measures selected by the alliances and the results of 2 key decisions made in constructing those measures.
PDF is available on the last page.