Currently Viewing:
The American Journal of Managed Care June 2017
Comparative Effectiveness and Costs of Insulin Pump Therapy for Diabetes
Ronald T. Ackermann, MD, MPH; Amisha Wallia, MD, MS; Raymond Kang, MA; Andrew Cooper, MPH; Theodore A. Prospect, FSA, MAAA; Lewis G. Sandy, MD, MBA; and Deneen Vojta, MD
Radical Prostatectomy Innovation and Outcomes at Military and Civilian Institutions
Jeffrey J. Leow, MBBS, MPH; Joel S. Weissman, PhD; Linda Kimsey, PhD; Andrew Hoburg, PhD; Lorens A. Helmchen, PhD; Wei Jiang, MS; Nathanael Hevelone, MPH; Stuart R. Lipsitz, ScD; Louis L. Nguyen, MD, MPH, MBA; and Steven L. Chang, MD, MS
Patients' Views of a Behavioral Intervention Including Financial Incentives
Judy A. Shea, PhD; Aderinola Adejare, BA; Kevin G. Volpp, MD, PhD; Andrea B. Troxel, ScD; Darra Finnerty, MPH; Karen Hoffer, BS; Thomas Isaac, MD, MPH, MBA; Meredith Rosenthal, PhD; Thomas D. Sequist, MD, MPH; and David A. Asch, MD, MBA
How Do Medicare Advantage Beneficiary Payments Vary With Tenure?
Paul D. Jacobs, PhD, and Eamon Molloy, PhD
Patient Ratings of Veterans Affairs and Affiliated Hospitals
Paul A. Heidenreich, MD, MS; Aimee Zapata, MS; Lisa Shieh, MD, PhD; Nancy Oliva, PhD, RN; and Anju Sahay, PhD
Currently Reading
Using "Roll-up" Measures in Healthcare Quality Reports: Perspectives of Report Sponsors and National Alliances
Jennifer L. Cerully, PhD; Steven C. Martino, PhD; Lise Rybowski, MBA; Melissa L. Finucane, PhD; Rachel Grob, PhD; Andrew M. Parker, PhD; Mark Schlesinger, PhD; Dale Shaller, MPA; and Grant Martsolf, PhD, MPH, RN
Patients' Views on Price Shopping and Price Transparency
Hannah L. Semigran, BA; Rebecca Gourevitch, MS; Anna D. Sinaiko, PhD; David Cowling, PhD; and Ateev Mehrotra, MD, MPH
Heterogeneity of Nonadherent Buprenorphine Patients: Subgroup Characteristics and Outcomes
Charles Ruetsch, PhD; Joseph Tkacz, MS; Vijay R. Nadipelli, MS, BPharm; Brenna L. Brady, PhD; Naoko Ronquest, PhD; Hyong Un, MD; and Joseph Volpicelli, MD, PhD
A Comparison of Retrospective Attribution Rules
Lucas Higuera, MA, and Caroline Carlin, PhD

Using "Roll-up" Measures in Healthcare Quality Reports: Perspectives of Report Sponsors and National Alliances

Jennifer L. Cerully, PhD; Steven C. Martino, PhD; Lise Rybowski, MBA; Melissa L. Finucane, PhD; Rachel Grob, PhD; Andrew M. Parker, PhD; Mark Schlesinger, PhD; Dale Shaller, MPA; and Grant Martsolf, PhD, MPH, RN
This study identifies practices and perceptions around public reporting of “roll-upâ€
ABSTRACT
 
Objectives: To understand the views of prominent organizations in the field of healthcare quality on the topic of reporting roll-up measures that combine indicators of multiple, often disparate, dimensions of care to consumers.  

Study Design: This study used a semi-structured, qualitative interview design.

Methods: We conducted 30- to 60-minute semi-structured telephone interviews with representatives of 10 organizations that sponsor public healthcare quality reports and 3 national alliances representing multiple stakeholder groups. We conducted a thematic analysis of interview transcriptions to identify common issues and concerns related to reporting roll-up measures.

Results: Among sponsors reporting roll-up measures, current practices for calculating and reporting these measures are diverse. The main perceived benefit of reporting roll-up measures is that they simplify large amounts of complex information for consumers. The main concern is the potential for consumers to misunderstand the measures and what associated roll-up scores communicate about provider performance. Report sponsors and national alliances feel that more guidance and research on the methods for producing and reporting scores for roll-up measures are needed.

Conclusions: The results of the interviews elucidate the need for research focused on construction and reporting of roll-up measures. Studies are needed to determine if roll-up measures are indeed perceived by consumers as being less complex and easier to understand.  
Takeaway Points
 
Some organizations that sponsor public reports of healthcare quality present roll-up measures that combine indicators of multiple, often disparate, dimensions of care. We found that: 
  • Report sponsors’ practices for calculating and reporting these measures are diverse. 
  • Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers but run the risk of misinterpretation. 
  • Report sponsors and national alliances indicate a need for more guidance and research on how to produce and report scores for roll-up measures.
A major objective of public reports on healthcare quality is to enable consumers to make well-informed choices about their care. To meet this objective, report sponsors often use strategies intended to reduce the cognitive burden of healthcare quality data and improve consumers’ comprehension and use of the information.1,2 One strategy is the provision of “roll-up” measures (ie, summary measures of healthcare quality that combine indicators of multiple, often disparate, dimensions of care). For example, a roll-up measure of patient experience with primary care could aggregate composite measures of patient–provider communication, access to care, and interactions with office staff. At a higher level of aggregation, a roll-up measure of overall hospital quality could combine measures of patient experience, clinical processes, and patient outcomes. The most prominent use of roll-up measures within healthcare is CMS’ Five-Star Quality Rating System, which assigns an overall rating of 1 to 5 stars to providers and plans based on multiple quality dimensions.3 Roll-up measures could also aggregate component measures at other levels. 

Publicly reporting roll-up measures offers advantages to consumers insofar as the roll-ups are created scientifically and grounded in high-quality research. These advantages include offering consumers a summary of multiple dimensions of care and, in many cases, a measure of healthcare quality that has superior reliability.4 However, this strategy is not without controversy:  roll-up measures may obscure important nuances of quality and not align with consumer preferences or information needs. Also, the calculation of roll-up scores relies on assumptions about the relative importance of each component measure.5-10

We aimed to understand how prominent organizations in the field of healthcare quality view the reporting of roll-up measures and obtain examples of current approaches to reporting roll-up measures. To that end, this study gathered information from representatives of report sponsors and national healthcare alliances on current practices and positive and negative perceptions with regard to creating and reporting roll-up measures to consumers. We sought to identify these organizations’ concerns given the current interest in reporting these measures to consumers and to determine the need for additional evidence to address these issues.

METHODS

In this qualitative study, we conducted 30- to 60-minute semi-structured telephone interviews with representatives from 10 organizations that publish comparative reports on healthcare quality (Table 1) and 3 national alliances that bring together multiple stakeholder groups. These alliances—the AQA (formerly the Ambulatory Care Quality Alliance), Consumer-Purchaser Alliance, and the National Quality Forum—were strategically selected for their expertise in healthcare quality measurement and reporting.

Interviews with report sponsors explored their perceptions of the benefits and drawbacks of reporting roll-up measures, the factors that influenced whether they report these measures, and for those that do report, how they calculate and report these measures to consumers. Interviews with the national alliances focused on whether the reporting of roll-up measures had been discussed within the organization, perceptions of the most important issues or concerns, the types of stakeholders involved in the discussion, and whether they created position statements or recommendations on reporting roll-up measures. Our qualitative approach enabled participants to describe their perceptions and experiences in their own words and was well-suited to eliciting participants’ diverse perspectives.11-13

We used thematic analysis to extract concepts that emerged from multiple interviews.14,15 Each interview transcription was reviewed by the first author (who also conducted several interviews with report sponsors and all 3 interviews with national alliances). The author extracted key information related to roll-up measure construction and reporting, identified consistent themes across the interviews, and produced short memos describing themes and the evidence for each. After clarifying assumptions and reviewing the themes with the other authors, the author updated the memos to reflect a consensus around the main themes extracted from the data. Key themes were confirmed through an independent review of transcriptions by the fourth author, who did not conduct any of the interviews.

RESULTS

Diversity of Roll-up Measure Reporting Practices

Among sponsors reporting roll-up measures at the time of the interviews, practices for calculating and reporting roll-up measures were diverse. Reporting practices for different providers varied in terms of construction and labeling (Table 2).16 For example, the Wisconsin Department of Employee Trust Funds differentially weighted and combined multiple Consumer Assessment of Healthcare Providers and Systems (CAHPS) and Healthcare Effectiveness Data and Information Set (HEDIS) indicators into a roll-up measure called the “Overall Quality Score” for health plans. In contrast, the Pacific Business Group on Health used an equal-weighting approach to combine several HEDIS measures of medical group performance into the measure, “Medical Group Provides Recommended Care.” 

In the absence of well-established methodological criteria, report sponsors based their decisions about how to calculate and label roll-up scores on information from a variety of sources, including research literature on reporting practices, technical expert guidance, information about other report sponsors’ practices, and the results of psychometric analyses. It is unclear exactly how they incorporated these sources of information into their choices about roll-up measure construction. Some participants reported testing measure labels with consumers or working with advisory groups to construct roll-up measures. All advisory groups included a diverse mix of stakeholders, such as purchasers, providers, state medical associations, and patient advocates. 

Simplification of Information

The main perceived benefit of reporting roll-up measures is the simplification of large amounts of complex information for consumers. Nearly all participants pointed out that roll-up measures make it easier for consumers to understand and use the information provided in quality reports. For example, one report sponsor representative said, “It’s a huge win for consumers to be able to go in and not have to read through 100 technical measures about hospitals. Even in plain language, 100 measures is a lot of stuff to expect people to go through.” Representatives of the national alliances suggested multiple ways in which reporting roll-up measures could reduce complexity for consumers. One representative asserted that processing roll-up measures requires less effort on the part of the consumer; another argued that reporting these measures allows consumers to gain a comprehensive view of care. In addition to reducing complexity, a third representative suggested that the reporting of roll-up measures could draw attention to the availability of comparative quality information, saying, “…to a large extent, the complexity of existing public reporting has meant that people don’t pay any attention and still ask the neighbors next door, ‘Who do you like?’…so you have to get it to the level that garners attention….” 

However, participants recognized that consumers vary in their need for information and some may want simplified information, but others desire detail (eg, the ability to drill down to scores for the measures that compose the roll-up measures). According to one national alliance representative, high-profile consumer groups argue that when consumers are shown roll-up measures, they should also have the option to view all component measures.

Understanding Roll-up Measures

The main concern about reporting roll-up measures is the potential for consumers to misunderstand the measures and what they communicate about provider quality. As one participant noted, if the weighting of individual scores in roll-up measures does not correspond to the relative importance of different aspects of care to consumers, consumers may draw incorrect conclusions about provider performance. Another participant reported that his organization opted against reporting a particular roll-up measure because of uncertainty about how to appropriately weight the component measures. Several report sponsors shared the concern about a potential mismatch between roll-up measures and the salience of individual measures to consumers. One sponsor suggested giving consumers a way to roll up information according to their preferences via an interactive online report.

Several other participants expressed concerns about consumers’ potential misunderstanding of roll-up measures and what they communicate. For example, consumers “might not read the fine print” and, therefore, might miss information about how the roll-up measures were calculated. They may also misunderstand a roll-up measure labeled as “overall quality,” as such a label may imply that all of the important aspects of quality have been measured and are thus represented by the score. In addition, these measures may obscure areas in which providers perform especially well or especially poorly. One report sponsor noted, “That’s always a risk—a person walking away with a generalized sense of performance, but it doesn’t encapsulate the variation of performance of the measures within the composite.”

Guidance and Research 

More guidance and research on the methods for producing and reporting scores for roll-up measures are needed. Several participants expressed a need for more evidence-based guidance on the construction and use of roll-up measures. In the words of one national alliance representative: “I think we could use more science of what logically hangs together, whether there are some things that should never be broken apart, whether there are some things that logically should always hang together.”

 

DISCUSSION


We conducted interviews with representatives of report sponsors and national alliances to document attitudes and practices around the calculation and reporting of roll-up measures to healthcare consumers. We identified diverse practices in the construction of roll-up scores; however, our study is limited by its small sample of strategically selected organizations, and we cannot determine the prevalence of roll-up score reporting. The interviews revealed a desire for more guidance and research on the methods for producing and reporting scores for roll-up measures. These findings point to the need for a research agenda focused on developing standards for the construction, calculation, and reporting of roll-up measures that are rooted in empirical data on how constructs assessed by various performance measures relate to each other and desired health outcomes.

 
Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
x
Welcome the the new and improved AJMC.com, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up