Dale Shaller, MPA

 To understand the views of prominent organizations in the field of healthcare quality on the topic of reporting roll-up measures that combine indicators of multiple, often disparate, dimensions of care to consumers.  

This study used a semi-structured, qualitative interview design.

We conducted 30- to 60-minute semi-structured telephone interviews with representatives of 10 organizations that sponsor public healthcare quality reports and 3 national alliances representing multiple stakeholder groups. We conducted a thematic analysis of interview transcriptions to identify common issues and concerns related to reporting roll-up measures.

Among sponsors reporting roll-up measures, current practices for calculating and reporting these measures are diverse. The main perceived benefit of reporting roll-up measures is that they simplify large amounts of complex information for consumers. The main concern is the potential for consumers to misunderstand the measures and what associated roll-up scores communicate about provider performance. Report sponsors and national alliances feel that more guidance and research on the methods for producing and reporting scores for roll-up measures are needed.

The results of the interviews elucidate the need for research focused on construction and reporting of roll-up measures. Studies are needed to determine if roll-up measures are indeed perceived by consumers as being less complex and easier to understand.  

Some organizations that sponsor public reports of healthcare quality present roll-up measures that combine indicators of multiple, often disparate, dimensions of care. We found that: 

Report sponsors’ practices for calculating and reporting these measures are diverse. 

Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers but run the risk of misinterpretation. 

Report sponsors and national alliances indicate a need for more guidance and research on how to produce and report scores for roll-up measures.

A major objective of public reports on healthcare quality is to enable consumers to make well-informed choices about their care. To meet this objective, report sponsors often use strategies intended to reduce the cognitive burden of healthcare quality data and improve consumers’ comprehension and use of the information.

 One strategy is the provision of “roll-up” measures (ie, summary measures of healthcare quality that combine indicators of multiple, often disparate, dimensions of care). For example, a roll-up measure of patient experience with primary care could aggregate composite measures of patient—provider communication, access to care, and interactions with office staff. At a higher level of aggregation, a roll-up measure of overall hospital quality could combine measures of patient experience, clinical processes, and patient outcomes. The most prominent use of roll-up measures within healthcare is CMS’ Five-Star Quality Rating System, which assigns an overall rating of 1 to 5 stars to providers and plans based on multiple quality dimensions.

 Roll-up measures could also aggregate component measures at other levels. 

Publicly reporting roll-up measures offers advantages to consumers insofar as the roll-ups are created scientifically and grounded in high-quality research. These advantages include offering consumers a summary of multiple dimensions of care and, in many cases, a measure of healthcare quality that has superior reliability.

 However, this strategy is not without controversy:  roll-up measures may obscure important nuances of quality and not align with consumer preferences or information needs. Also, the calculation of roll-up scores relies on assumptions about the relative importance of each component measure.

We aimed to understand how prominent organizations in the field of healthcare quality view the reporting of roll-up measures and obtain examples of current approaches to reporting roll-up measures. To that end, this study gathered information from representatives of report sponsors and national healthcare alliances on current practices and positive and negative perceptions with regard to creating and reporting roll-up measures to consumers. We sought to identify these organizations’ concerns given the current interest in reporting these measures to consumers and to determine the need for additional evidence to address these issues.

In this qualitative study, we conducted 30- to 60-minute semi-structured telephone interviews with representatives from 10 organizations that publish comparative reports on healthcare quality (Table 1) and 3 national alliances that bring together multiple stakeholder groups. These alliances—the AQA (formerly the Ambulatory Care Quality Alliance), Consumer-Purchaser Alliance, and the National Quality Forum&mdash;were strategically selected for their expertise in healthcare quality measurement and reporting.

Interviews with report sponsors explored their perceptions of the benefits and drawbacks of reporting roll-up measures, the factors that influenced whether they report these measures, and for those that do report, how they calculate and report these measures to consumers. Interviews with the national alliances focused on whether the reporting of roll-up measures had been discussed within the organization, perceptions of the most important issues or concerns, the types of stakeholders involved in the discussion, and whether they created position statements or recommendations on reporting roll-up measures. Our qualitative approach enabled participants to describe their perceptions and experiences in their own words and was well-suited to eliciting participants’ diverse perspectives.

We used thematic analysis to extract concepts that emerged from multiple interviews.

 Each interview transcription was reviewed by the first author (who also conducted several interviews with report sponsors and all 3 interviews with national alliances). The author extracted key information related to roll-up measure construction and reporting, identified consistent themes across the interviews, and produced short memos describing themes and the evidence for each. After clarifying assumptions and reviewing the themes with the other authors, the author updated the memos to reflect a consensus around the main themes extracted from the data. Key themes were confirmed through an independent review of transcriptions by the fourth author, who did not conduct any of the interviews.

Diversity of Roll-up Measure Reporting Practices

Among sponsors reporting roll-up measures at the time of the interviews, practices for calculating and reporting roll-up measures were diverse. Reporting practices for different providers varied in terms of construction and labeling (Table 2).

 For example, the Wisconsin Department of Employee Trust Funds differentially weighted and combined multiple Consumer Assessment of Healthcare Providers and Systems (CAHPS) and Healthcare Effectiveness Data and Information Set (HEDIS) indicators into a roll-up measure called the “Overall Quality Score” for health plans. In contrast, the Pacific Business Group on Health used an equal-weighting approach to combine several HEDIS measures of medical group performance into the measure, “Medical Group Provides Recommended Care.” 

In the absence of well-established methodological criteria, report sponsors based their decisions about how to calculate and label roll-up scores on information from a variety of sources, including research literature on reporting practices, technical expert guidance, information about other report sponsors’ practices, and the results of psychometric analyses. It is unclear exactly how they incorporated these sources of information into their choices about roll-up measure construction. Some participants reported testing measure labels with consumers or working with advisory groups to construct roll-up measures. All advisory groups included a diverse mix of stakeholders, such as purchasers, providers, state medical associations, and patient advocates. 

The main perceived benefit of reporting roll-up measures is the simplification of large amounts of complex information for consumers. Nearly all participants pointed out that roll-up measures make it easier for consumers to understand and use the information provided in quality reports. For example, one report sponsor representative said, “It’s a huge win for consumers to be able to go in and not have to read through 100 technical measures about hospitals. Even in plain language, 100 measures is a lot of stuff to expect people to go through.” Representatives of the national alliances suggested multiple ways in which reporting roll-up measures could reduce complexity for consumers. One representative asserted that processing roll-up measures requires less effort on the part of the consumer; another argued that reporting these measures allows consumers to gain a comprehensive view of care. In addition to reducing complexity, a third representative suggested that the reporting of roll-up measures could draw attention to the availability of comparative quality information, saying, “…to a large extent, the complexity of existing public reporting has meant that people don’t pay any attention and still ask the neighbors next door, ‘Who do you like?’…so you have to get it to the level that garners attention….” 

However, participants recognized that consumers vary in their need for information and some may want simplified information, but others desire detail (eg, the ability to drill down to scores for the measures that compose the roll-up measures). According to one national alliance representative, high-profile consumer groups argue that when consumers are shown roll-up measures, they should also have the option to view all component measures.

The main concern about reporting roll-up measures is the potential for consumers to misunderstand the measures and what they communicate about provider quality. As one participant noted, if the weighting of individual scores in roll-up measures does not correspond to the relative importance of different aspects of care to consumers, consumers may draw incorrect conclusions about provider performance. Another participant reported that his organization opted against reporting a particular roll-up measure because of uncertainty about how to appropriately weight the component measures. Several report sponsors shared the concern about a potential mismatch between roll-up measures and the salience of individual measures to consumers. One sponsor suggested giving consumers a way to roll up information according to their preferences via an interactive online report.

Several other participants expressed concerns about consumers’ potential misunderstanding of roll-up measures and what they communicate. For example, consumers “might not read the fine print” and, therefore, might miss information about how the roll-up measures were calculated. They may also misunderstand a roll-up measure labeled as “overall quality,” as such a label may imply that all of the important aspects of quality have been measured and are thus represented by the score. In addition, these measures may obscure areas in which providers perform especially well or especially poorly. One report sponsor noted, “That’s always a risk—a person walking away with a generalized sense of performance, but it doesn’t encapsulate the variation of performance of the measures within the composite.”

More guidance and research on the methods for producing and reporting scores for roll-up measures are needed. Several participants expressed a need for more evidence-based guidance on the construction and use of roll-up measures. In the words of one national alliance representative: “I think we could use more science of what logically hangs together, whether there are some things that should never be broken apart, whether there are some things that logically should always hang together.”

We conducted interviews with representatives of report sponsors and national alliances to document attitudes and practices around the calculation and reporting of roll-up measures to healthcare consumers. We identified diverse practices in the construction of roll-up scores; however, our study is limited by its small sample of strategically selected organizations, and we cannot determine the prevalence of roll-up score reporting. The interviews revealed a desire for more guidance and research on the methods for producing and reporting scores for roll-up measures. These findings point to the need for a research agenda focused on developing standards for the construction, calculation, and reporting of roll-up measures that are rooted in empirical data on how constructs assessed by various performance measures relate to each other and desired health outcomes.

One key topic for exploration is the appropriate level of aggregation for roll-up measures. Roll-ups could aggregate measures from a single source (eg, the CAHPS Clinician & Group Survey) or a single domain (eg, clinical quality, patient experience). Alternatively, a single overall roll-up measure, such as the star ratings provided through the CMS Five-Star Rating System, could encompass multiple domains of quality. Although this type of measure may maximize simplicity for consumers, constructing such a score requires strong and nontrivial assumptions about how best to reliably and accurately use a single indicator to communicate the multidimensional construct of healthcare quality.

Although participants perceived that reporting roll-up scores to consumers helps simplify the complex array of healthcare quality information presented in reports, they were concerned about the potential for consumers to misunderstand the aspects of quality captured by roll-up measures and how providers perform on discrete aspects of quality. Whether these beliefs are well-founded is a question worth investigating. Some report sponsors conduct research on roll-up measure construction,

 but the body of published research on consumer use of, and responses to, roll-up measures is minimal.

Consumers are used to seeing roll-ups in other realms (eg, 

 product reviews), but reporting roll-up measures of healthcare quality is relatively new. It is unclear whether consumers find these measures to be easier to understand than the component measures from which they are constructed or how these measures are interpreted. It is also unclear how viewing roll-up measures affects consumers’ choices of plans and providers, what consumers interpret rolled-up measures to signify, or whether roll-up measures simplify available data to the point where differences in performance no longer seem meaningful. Finally, little is known about whether roll-ups might deter crucial forms of consumer learning about performance metrics and provider variation in quality.

Consumer interpretation and use of roll-up measures likely vary depending on the level of aggregation of scores. The benefits and drawbacks of other stakeholders’ use of roll-up measures (eg, providers’ potential use of roll-ups for quality improvement purposes) were not documented here given the focus on report sponsors’ expectations about consumers’ use of roll-up scores; however, these expectations are important to understand. Because the reporting of roll-up measures is underway at federal agencies and among some report sponsors, it is critical to better understand how consumers perceive and respond to such measures. 

We found that report sponsors’ practices for calculating and reporting these measures are diverse. Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers, but run the risk of misinterpretation. The results of the study elucidate the need for research focused on the construction and reporting of roll-up measures, as well as consumer interpretations of roll-up measures.

 RAND Corporation (JLC, SCM, MLF, AMP, GM), Pittsburgh, PA; Severyn Group (LR), Ashburn, VA; University of Wisconsin (RG), Madison, WI; Yale University (MS), New Haven, CT; Shaller Consulting Group (DS), Stillwater, MN.

 This paper was supported by 2 cooperative agreements (2U18HS016980 and 1U18HS016978) from AHRQ to RAND Corporation and Yale University, respectively.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (JLC, RG, GM, SCM, AMP, LR, DS, MS); acquisition of data (JLC, RG, GM, SCM, LR); analysis and interpretation of data (JLC, MLF, RG, GM, SCM, LR, DS, MS); drafting of the manuscript (JLC, MLF, AMP, MS); critical revision of the manuscript for important intellectual content (JLC, MLF, RG, GM, SCM, AMP, LR, DS); statistical analysis (JLC, MLF).

 Jennifer L. Cerully, PhD, RAND Corporation, 4570 Fifth Ave, Ste 600, Pittsburgh, PA 15213. E-mail: jcerully@rand.org.

1. Schlesinger M, Kanouse DE, Rybowski L, Martino SC, Shaller D. Consumer response to patient experience measures in complex information environments. 

. 2012;50(suppl):S56-S64. doi: 10.1097/MLR.0b013e31826c84e1.

2. Hibbard JH, Slovic P, Peters E, Finucane ML. Strategies for reporting health plan performance information to consumers: evidence from controlled studies. 

3. Hospital Compare star ratings fact sheet. CMS website. https://www.cms.gov/Newsroom/MediaReleaseDatabase/Fact-sheets/2015-Fact-sheets-items/2015-04-16.html. Published April 16, 2015. Accessed October 20, 2015.

4. Zaslavsky AM, Shaul JA, Zaborski LB, Cioffi MJ, Cleary PD. Combining health plan performance indicators into simpler composite measures. 

5. Romano P, Hussey P, Ritley D. Selecting quality and resource use measures: a decision guide for community quality collaboratives. Agency for Healthcare Research and Quality website. https://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/perfmeasguide/index.html. Published May 2010. Accessed January 26, 2016.

6. Martsolf GR, Scanlon DP, Christianson JB. Multistakeholder perspectives on composite measures of ambulatory care quality: a qualitative descriptive study. 

. 2013;70(4):434-448. doi: 10.1177/1077558713485134.

7. Boyce T, Dixon A, Fasolo B, Reutskaja E. Choosing a high quality hospital: the role of nudges, scorecard design and information. The King’s Fund website. https://www.kingsfund.org.uk/sites/files/kf/field/field_publication_file/Choosing-high-quality-hospital-role-report-Tammy-Boyce-Anna-Dixon-November2010.pdf. Published November 2010. Accessed January 26, 2016.

8. Guiding principles for public reporting of provider performance. Association of American Medical Colleges website. https://www.aamc.org/download/370236/data/guidingprinciplesforpublicreporting.pdf. Published March 2014. Accessed May 27, 2015.

9. Orlowski JM. Re: AAMC comments on the measure selection for Hospital Compare Star Ratings TEP Report. Association of American Medical Colleges website. https://www.aamc.org/download/425936/data/aamccommentletteroncmsstarratingstep.pdf. Published February 25, 2015. Accessed May 27, 2015.

10. Thompson A. Re: comments on the Hospital Star Ratings methodology report. American Hospital Association website. http://www.aha.org/advocacy-issues/letter/2015/150914-cl-StarRatings.pdf. Published September 14, 2015. Accessed October 20, 2015.

11. Pope C, Mays N. Qualitative research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. 

12. Morgan DL, ed. Successful Focus Groups: Advancing the State of the Art. Newbury Park, CA: SAGE Publications; 1993.

13. O’Brien K. Improving survey questionnaires through focus groups. In: Morgan DL, ed. 

Successful Focus Groups: Advancing the State of the Art

. Newbury Park, CA: SAGE Publications; 1993: 105-118.

14. Boyatzis RE. Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks, CA: SAGE Publications; 1998.

15. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-101.

16. Five-Star Quality Rating System. CMS website. www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/FSQRS.html. Updated March 22, 2017. Accessed March 3, 2014.

Articles

This study identifies practices and perceptions around public reporting of â€œroll-upâ€

Using "Roll-up" Measures in Healthcare Quality Reports: Perspectives of Report Sponsors and National Alliances

Whether incentive programs lead to more effective improvement activities by medical practices remains an open question. We assessed the impact of a pay-for-performance program designed to improve patients’ experiences with primary care and identified the factors that influenced the outcomes.

We conducted telephone interviews with clinical and administrative leaders of 8 physician practices that had been identified by a major health insurance plan as needing to improve on their past scores on a patient experience survey.

 The financial incentive had almost no effect on the priorities or activities of the medical practices. Few practice leaders were aware of the incentive. All were familiar with local efforts to improve patients’ experiences and some maintained that focus over time, but not because of the ongoing system-level initiative. The effectiveness of the incentive program was limited by the program design, the improvement goals, the timing of progress measurement, and a lack of ongoing communication about the program’s purpose, available support, and goals. Medical practice leaders’ understanding of the program and its goals was limited; providers infrequently used the free educational programs available; and other initiatives may have taken priority.

 These findings suggest that if system-level incentives are to improve care quality, they must be designed carefully to reach the audience responsible for improving care, to motivate organizational change, and to support ongoing communications with practice leaders.Both public and private payers have been expanding the use of pay-for-performance (P4P) programs that encourage improvements in primary care by linking financial rewards to evidence of higher quality. These programs have evolved in response to concerns that many weaknesses in the healthcare system are a result of the way providers are compensated. Whether P4P programs lead to improvements in the quality of healthcare remains an open question. There are relatively little data on the influence of such programs

 and the available evidence on their effectiveness is mixed.

Studies that show an impact have found modest effects

; these studies have focused on measures of clinical process and intermediate outcomes.

Given the increasing attention to patient experience measures in value-based purchasing programs, such as the CMS Medicare Shared Savings Program,

it is important to understand the effectiveness of P4P strategies in driving improvements in patient experience.

The purpose of this study was to assess whether and how an incentive program for a large provider network to improve primary care patients’ experiences affected the priorities and activities of medical practices. In this article, we describe the program and use the framework proposed by Van Herck and colleagues to assess several factors that might have affected its results.

The review of 128 evaluation studies by Van Herck and colleagues concluded that the impact of P4P programs can be influenced by the program’s context as well as various design choices, including the ways in which quality is measured, the quality goals and target, the nature of the incentives, how the program is implemented and communicated, and how the effects are evaluated. This framework offers useful insights into the factors that may have undermined or facilitated the effectiveness of the P4P program in driving improvements in patient experience.

In 2004, a large healthcare network negotiating with a major health insurance plan in its market

 agreed to a 5-year P4P program that would financially reward practices for attaining specified goals. An improvement in patients’ experiences with primary care was one of the performance goals for the second phase of the contract (2007-2009). The incentive was that part of a “revenue withhold” would be returned to the network if lower-performing primary care practices improved their scores on a standardized patient experience survey adapted from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician & Group Survey.

The program included 96 primary care practice sites in the network that scored below the statewide mean on 1 or more measures of patient experience. The total value of the withhold tied to patient experience was about $5 million.

To help the low-performing practices improve patient experience, the network contracted with an internal center with expertise in primary care improvement. The center supported the practices in developing priorities and improvement plans and offered free educational and consulting services designed to help the practices improve the domains of patient experience addressed by the survey.

Under the terms of the P4P contract, the network was required to submit improvement plans for 95% of low-performing practices. In the third year, the practices were expected to achieve specified improvements in performance. The percentage of the withhold that would be available to the practices ranged from 25% to 100%, depending on the percentage of practices that met their target based on the 2009 survey results. If fewer than 55% of practices met the target, none of the withheld revenue would be returned.

In 2008, 2 of the authors (LR and DS) conducted in-person and telephone interviews with several individuals involved in the P4P program from the health plan, the provider network, the internal consulting center, and an independent organization that gathers and reports the survey results. These interviews elicited information about the P4P program as well as perspectives on the program’s development, purpose, and feasibility.

The 96 physician practices that met the criteria for participation in the P4P program were affiliated with 16 large organizations that manage the practices. To select potential interviewees for this study, we first identified the 2 survey topics chosen as improvement priorities by the most practices: doctor-patient communication and office staff courtesy and respect. Our sample consisted of the 31 practices that chose either of the 2 topics; these practices represented 11 of the large physician organizations.

The 31 practices were divided into 2 groups: those that met the target (16) and those that did not (15). The original goal was to interview 2 practices that met the goal and 2 that did not for each of the 2 topics. However, it was not possible to identify practices in all 4 categories because all of the practices that focused on communication met their target. Thus, we selected practices that represented a mix of topics (ie, both communication and office staff), a mix of results for the office staff measure, and a mix of large physician organizations.

The study protocol was approved by the Yale Human Investigations Committee. All subjects knew that we were conducting research and agreed to be interviewed.

In 2011, we invited 17 practices by e-mail and phone to participate in 30- to 60-minute telephone interviews. If a practice declined to participate, another practice was selected and contacted. In total, the authors interviewed clinical and administrative leaders of 8 medical practices, 1 of which included multiple sites. These sites represented approximately one-third of the medical practices focusing on the 2 topics: 11 of the 31 practice sites and 5 of the 16 large physician organizations.

The interviews with the clinical and administrative practice leaders revealed a low level of awareness of the P4P program. Only 1 of the interviewees knew that the practice had a financial incentive to improve patient experience scores; none was aware that the network did not receive the withheld dollars tied to patient experience because slightly fewer than 55% of the participating practices had met their performance target.

Moreover, the practice leaders were not aware that the goal of improving patient experience was part of an ongoing network initiative. While they knew about the network’s initial efforts to focus their attention on patient experience, those that maintained that focus over time did not characterize their efforts as part of a systemwide initiative to achieve specified goals.

While all the practices were aware of the public report of patient experience survey results, several had never looked at the report and almost none had used it. To assess their performance, they relied on their own survey results or those collected for their practice by the provider organizations with which they were affiliated. The differences between the publicly reported measures and scores and those from other sources caused confusion and frustration.

The practice leaders also varied significantly in their awareness, understanding, and use of the free educational and consulting services. While some took full advantage of the offerings, others were either unaware of the services or uninterested. Also, even though several practice leaders indicated that doctors or staff had participated in and benefited from at least 1 educational event, not all of them were clear on what organization sponsored the events or that the services were related to the incentive to improve patient experience.

Because of the requirement to produce improvement plans, the P4P program initially succeeded in capturing the practices’ attention. However, the long-term incentive generally did not lead to an ongoing focus on patient experience. While a few practices continued to plan and implement improvement strategies, most turned their attention to other challenges, noting that patient experience was only one of many issues that require their attention and resources.

Categorizing the interview comments according to the framework of Van Herck and colleagues

 suggests why the P4P program failed to bring about a sustained effort to improve patient experience.

The network-level incentive to improve patient experience took place in a context that placed conflicting demands on practices. First, this program coincided with a statewide health reform effort that resulted in insurance coverage for most residents. For many practices, the subsequent pressure to meet the demand for primary care services overwhelmed their ability to focus on other needs. All of the practices noted that improving patients’ access to care and information had been one of their highest priorities in recent years. While access to care is an element of patient-centered care, it was not the domain that these practices had committed to improve under the P4P program. Thus, to some extent, the practices’ commitment to improving one aspect of patient experience was competing with the urgent need to improve another aspect.

Constant competition for the attention of administrative and clinical leaders was also a key part of the external context. Within the network, the goal to improve patient experience was just 1 of several P4P goals. When measures and goals change from year to year, it is a significant challenge for healthcare leaders to maintain a focus on any improvement target over time.

 In addition, other health plans and payers were seeking improvements in specific areas. Health plans in the state had been implementing performance incentives for several years: a 2004 survey found that 89% of the medical groups had a P4P incentive in at least 1 commercial plan contract. Over a third of the surveyed groups reported that their incentives were tied to their performance on patient satisfaction surveys.

Finally, organizational identity and affiliation contributed to the context for the participating medical practices.

None of the interviewees said that that their improvement work was focused on a goal shared with primary care practices across the provider network. To the extent that they referred to anything other than their own practices, it was their affiliation with a large health system or medical group. However, the goals of the network’s P4P contract were not always aligned with the goals and programs of these other, more proximate organizations.

Friedberg et al found that a majority of medical groups in the study state were working to improve patient experience: 61% reported groupwide improvement efforts and 22% were focused on improving care from low-scoring physicians or practice sites.

 Those groups were more likely than other groups to have some financial incentive to improve. Thus, it is likely that at least some of the practices affected by the network-level program were also participating in a group-level program. However, the improvement targets and strategies for the medical groups were not identical. Although both programs focused on communication, for example, the improvement initiatives at the group level did not emphasize doctor-patient communication; they were primarily focused on organizational factors, such as redesigning office work flow, training nonclinicians, using electronic health records, and reassigning staff responsibilities.

The measures used to assess practice performance are a key element of a P4P program. While consumers and healthcare providers have had access to reports with results of the CAHPS Clinician & Group Survey since 2006, the interviews revealed that not everyone accepted the survey scores as an accurate reflection of quality. Some practice leaders noted that a small subset of their physicians were skeptical that quality as measured by the patient experience survey was even a problem. Others accepted the measures but expressed defensiveness about their performance, claiming that the results were not adequately adjusted to reflect differences in patient populations. This lack of trust in the measures may have undermined efforts to improve quality.

One positive element of this program was that its focus on practices with a baseline performance below the 2007 mean meant that all participants had room to improve. Because of the timing of the surveys, however, it was not possible to assess the degree of improvement until the results of the 2009 survey had been analyzed. Practices that initiated changes to improve patient experiences in 2008 had to wait until the spring of 2010 to find out whether they achieved their targets on the 2009 survey.

More frequent assessments and payments (eg, quarterly bonuses) might have served as an important reminder of the P4P program and its goals.10 Some practices tried to set interim goals by examining regular feedback based on other patient experience surveys, but found that they could not always relate the measures in those surveys to the measures in the statewide survey.

Van Herck et al found that the incentive size for many P4P programs in the United States is low (estimated at 1% to 2% of income) and that there was no relationship between incentive size and effect.

 The issue for this program was not the size of the incentive but the lack of clarity about how much money was at stake for a practice or an individual physician. The importance of targeting an incentive at the individual and/or team level is well established.

 However, the provider network allowed the large organizations that manage groups of practices to determine the flow of funds from P4P activities. Some kept all P4P money they earned to support the organization while others shared the funds with the practices. Because of variation in the way incentives were distributed, no practice knew what the incentive was worth to them, and only some practices offered incentives to individual physicians.

Furthermore, although the unit of analysis for determining progress was the primary care practice, whether an incentive was earned depended on the combined performance of a group of low-performing practices, which was a subset of all primary care practices in the network. Practices that succeeded in reaching their own target did not get anything if the entire group did not make its overall target. It is possible that this “shared benefit” design led some practices to assume that they did not have to put much effort into improving on the selected measure because everyone else would do what was needed to gain the incentive.

Implementing and Communicating the Program

Another factor that may have undermined the effect of the P4P program was that once medical practices had submitted improvement plans, they were not required to do anything else, including developing the capacity to manage and conduct improvement work.

Among the interviewed practices, none had a dedicated quality improvement team and only a few had someone in a leadership role with expertise in quality improvement methods.

The limited awareness of this program points to a failure to communicate with the participating providers, which has been associated with the lack of an effect in other P4P programs.

 The network leadership did not continue communicating and emphasizing the importance of the program’s goals. With the exception of 2 newsletter articles, the practices did not receive regular communications about this program to reinforce its goals and rationale, to explain how the program worked, or to hold practice leadership accountable. The center under contract to support the practices had contact with the practices, but the staff did not have access to established communication systems, nor were they allowed to contact the practice administrators directly. The provider network required that they communicate with the practices through the network’s account managers.

Individuals from about half of the study practices indicated that someone in the practice took advantage of the training and other resources provided by the center tasked with supporting the practices. However, there were some misunderstandings and confusion about the nature and scope of the resources. Several interviewees seemed unsure about the types of resources that were available and questioned whether the resources were applicable to the issues they needed to address. They also expressed confusion about who was offering the resources—even if they had already attended a training session&mdash;and whether they could or should take advantage of the offerings, which appeared to some to be offered by a rival hospital system or to be redundant with other services.

As noted above, there was a significant gap in time between the practices’ initial efforts and the reporting of results from the 2009 survey, which made it difficult for the practices to gauge whether they were making progress. Furthermore, the results from the 2007 survey were not available until April 2008, and the improvement plans had to be finalized by September 2008. The time between the implementation of those plans and the follow-up measurement in the fall of 2009 may not have been sufficient to assess the impact on patient experience.

This study points to several factors that have the potential to limit the effectiveness of incentives. The first is the flow of money (ie, whether the financial reward actually accrues to the organizational units or individuals responsible for improvements in care). The second factor is the nature of communication among the different levels of a large healthcare organization. In the absence of clear and ongoing communication about the incentive, there was no shared understanding of the goals or who stood to gain or lose, thus limiting the effect of the incentive. Insufficient communication also meant that practices were not aware of and made little use of the educational and consulting support available to them.

This study also suggests that P4P programs may benefit from some coordination with other similar quality improvement programs in the region—or at least efforts to ensure that the goals will not conflict with those of other initiatives. In this case, the primacy of improving access to primary care may have taken precedence over other patient experience goals.

Two potential limitations of this study were the small number of practices that we studied and the timing of the interviews. The responses of the selected practices, about 10% of the total number of practices in the P4P program, may not be representative of all other practices. Timing may be an issue because the interviews with practice leaders took place more than a year after the pay-for-performance program was initiated. In many cases, the leaders had moved on to new goals and did not necessarily remember what they had done to meet their improvement target, when they had done it, or why. Nevertheless, the results reported here support our conclusion that most practices were not taking a long-term approach to addressing the performance issues associated with the incentive.

The use of system-level incentives as described in this study was not an effective strategy for encouraging improvements in patient experience with care. Future efforts to motivate improvement across a health system may benefit from a more direct connection of the financial reward or penalty to those responsible for improving care, clear and ongoing communications with clinical and administrative leaders, and better coordination among the organizations that set improvement goals for physician practices.

The Severyn Group (LR), Ashburn, VA; Shaller Consulting Group (DS), Stillwater, MN; John D. Stoeckle Center for Primary Care Innovation, Massachusetts General Hospital (SEL), Boston; Yale School of Public Health (PDC), New Haven, CT.

Work on this project was supported by a cooperative agreement from the Agency for Healthcare Research and Quality (#U18HS016978).

 Concept and design (LR, DS, PDC, SEL); acquisition of data (LR, DS, SE-L); analysis and interpretation of data (LR, DS); drafting of the manuscript (LR, SE-L); critical revision of the manuscript for important intellectual content (DS, PDC); provision of patients or study materials (SE-L); obtaining funding (PDC); administrative, technical, or logistic support (SE-L); and supervision (SE-L).

Multiple factors can impact the effectiveness of financial incentives intended to encourage primary care providers to improve patient experiences.

Dale Shaller, MPA

Articles

Using "Roll-up" Measures in Healthcare Quality Reports: Perspectives of Report Sponsors and National Alliances

Impact of Incentives to Improve Care for Primary Care Patients