Continuous subcutaneous insulin infusion (CSII), or “insulin pump” therapy, is an alternative to multiple daily insulin injections (MDII) for management of diabetes. This study evaluates patterns of healthcare utilization, costs, and blood glucose control for patients with diabetes who initiate CSII.

 Pre-post with propensity-matched comparison design involving commercially insured US adults (aged 18-64 years) with insulin-requiring diabetes who transitioned from MDII to CSII between July 1, 2009, and June 30, 2012 (“CSII initiators”; n = 2539), or who continued using MDI (n = 2539).

 Medical claims and laboratory results files obtained from a large US-wide health payer were used to construct direct medical expenditures, hospital use, healthcare encounters for hypoglycemia, and mean concentration of glycated hemoglobin (A1C). We fit difference-in-differences regression models to compare healthcare expenditures for 3 years following the switch to CSII. Stratified analyses were performed for prespecified patient subgroups.

Over 3 years, mean per-person total healthcare expenditures were $1714 (95% confidence interval [CI], $1184-$2244) higher per quarter for CSII initiators compared with matched MDII patients (total mean 3-year difference of $20,565). Compared with matched controls, mean A1C concentrations became lower for CSII initiators by 0.46% in year 2 (

= .047). CSII initiators also had a higher rate of hypoglycemia encounters in year 1 (

For adults with insulin-requiring diabetes, transitioning from MDII to CSII was associated with modest improvements in A1C but more hypoglycemia encounters and increased healthcare expenditures, without significant improvement in other potentially offsetting areas of healthcare consumption.

Mean total healthcare expenditures were estimated to be $1714 higher per person per quarter for patients who initiate continuous subcutaneous insulin infusion (CSII) compared with matched multiple daily insulin injection (MDI) patients, equating to a total 3-year mean difference of about $20,565 per person. 

Subgroup analyses showed modest differences in glycated hemoglobin favoring CSII recipients in years 2 and 3, but also statistically significant differences in healthcare visits for hypoglycemia as CSII was being initiated. 

It may remain challenging for health payers or providers to develop policies regarding access or coverage for CSII when, in aggregate, CSII appears to add immediate costs with short-term benefits that are uncertain and/or are difficult to measure.

Diabetes affects 29 million Americans, imposing a substantial health and economic burdens on the US population.

 Intensive blood glucose management improves health outcomes for most patients,

 but achieving this goal requires them to monitor blood glucose more often, use more medications, including insulin; and to follow up regularly with healthcare providers. These activities increase healthcare expenditures and are perceived by many patients to reduce their quality of life (QOL).

 This has stimulated efforts to develop less demanding forms of insulin delivery to reduce patient burden, while attempting to limit adverse effects, such as hypoglycemia and weight gain.

Over the past 2 decades, there has been an increase in the use of continuous subcutaneous insulin infusion (CSII) therapy, or “insulin pumps,” for the treatment of diabetes.

 An insulin pump is a small programmable device that uses a computer algorithm to administer insulin both continuously and by patient-initiated bolus infusion through a small catheter that is left under the skin. The American Diabetes Association and American Association of Clinical Endocrinologists recommend intensive blood glucose management using either multiple daily insulin injections (MDII) or CSII, but advise CSII primarily for well-educated and motivated patients who are unable to achieve optimal glycemic control with MDII.

Prior research has suggested some potential advantages of CSII over MDII therapy, such as reduced pain, less stigma, lower frequency and severity of hypoglycemia, and overcoming some barriers to adherence.

 However, a recent meta-analysis of trials directly comparing MDII with CSII found that CSII did not reduce hypoglycemia or weight gain and had inconsistent effects on QOL and blood glucose control, with the results of only 1 trial showing greater improvement in glycated hemoglobin (A1C) among adults with type 1 diabetes (T1D) who had comparatively higher A1C levels at baseline.

 Recent reports have called for clearer evidence-based guidelines for CSII indications and more transparent reporting of safety data by pump manufacturers.

Insulin infusion pumps have been reported to cost about $4500, with additional costs for supplies exceeding $1500 per person per year.

 Because health insurance typically pays for 80% to 90% of CSII costs, access to insurance and coverage policies naturally play a strong role in whether patients choose to initiate this approach. Health payers generally offer insurance coverage for CSII to patients who meet specific clinical criteria and demonstrate good self-management practices and participation with healthcare visits. Medicare provides coverage for CSII in patients who have evidence of either: a) prior use of CSII with adherence to glucose self-testing prior to Medicare enrollment, or b) persistent hyperglycemia, recurring hypoglycemia, or other forms of poor blood glucose control despite completion of a comprehensive diabetes education program and adherence to glucose self-monitoring and MDII for at least 6 months.

 Patients must then complete regular follow-up visits with a supervising physician at least every 3 months to continue receiving health payer coverage for CSII.

Although prior research has demonstrated the high acquisition costs for insulin infusion pumps and supplies, it is not yet known if those higher upfront costs can be recovered through improvements in health that translate into higher QOL and/or lower utilization of healthcare services. A related important question is whether those outcomes might vary among different subgroups of patients.

 To address these gaps, we designed this study to evaluate patterns of healthcare utilization and costs for adults who primarily have T1D and transition to CSII in practice, comparing them with otherwise similar patients who continued MDII therapy.

We used a pre-post with propensity-matched comparison group quasi-experimental study design with difference-in-differences estimation to evaluate patterns of healthcare costs, utilization, and blood glucose control associated with the initiation of CSII by adults with insulin-requiring diabetes who received health insurance coverage from a single, large, nationwide commercial health insurer. The study design is depicted in Figure 1 and described further below.

The evaluation sample included patients with insulin-requiring diabetes who were aged 18 to 64 years. Insulin-requiring diabetes was defined as at least 1 encounter with an 

International Classification of Diseases, Ninth Revision, Clinical Modification 

) code 250.XX and at least 1 dispensing event for MDII, but no previous use of an oral hypoglycemic class medication. This exclusion made it more likely, but not certain, that individuals included in the study had T1D. CSII initiators were defined by a new dispensing event for an infusion pump (Healthcare Common Procedure Coding System billing code E0784 or A9274) any time between July 1, 2009, and June 30, 2012. Additional CSII-related utilization and supplies were identified using specific procedural and encounter codes (see eAppendix Table 1 for details [

]). The date of the first occurrence of one of these codes within 270 days before the pump device dispensing date was defined as the index date for our pre-post analysis (Figure 1). This date occurred a median of 145 days before the pump dispensing date, and the time between these 2 dates was referred to as the “pump onboarding” period. In follow-up analyses, CSII initiators were considered to have stopped CSII if they had 120 days of enrollment without any claims for CSII-related supplies or services and had at least 2 basal insulin refills during that same period.

To identify a comparison group with similar baseline characteristics, we used a nearest-neighbor propensity-matching approach.

 The patient-level propensity model used logistic regression to predict the odds of CSII initiation and included baseline variables relating to the outcomes of interest, for which prior research results have demonstrated associations with diabetes treatment choice or intensity. Propensity model predictive variables included patient sex, age category, Charlson Comorbidity Index score, presence/absence of a prior obesity-related 

 code, most recent A1C test result ≥ or <8% (or “value missing” if not available), and each of the following utilization criteria within 180 days before the CSII index date: total encounters for a hypoglycemia diagnosis, total encounters with an endocrinologist, and the presence or absence of at least 1 billing code suggestive of poorly controlled diabetes (eAppendix Table 1). To enable direct comparisons between groups, matched control clients were assigned the same index date as the CSII initiator to whom they were paired. To construct baseline variables for the match, individuals in both groups were required to have continuous health plan enrollment for at least 3 months before and 3 months after the index date.

Study outcomes included direct medical costs, categorized as inpatient, outpatient, pharmacy, and total healthcare costs; emergency department (ED) visits; encounters for hypoglycemic events; and, when available, A1C values. Total healthcare costs, including both health plan and patient cost share components, were assessed equally across all patient groups by applying a standardized price for each claim. To minimize the effect of extreme outliers on mean cost estimates, we replaced costs above the 95th percentile with the 95th percentile value.

Data sources included national member enrollment files, medical inpatient and ambulatory claims, and pharmacy claims made available by a large US-based commercial health insurer. Although the completion of all laboratory tests (including A1C tests) could be determined for all patients in both groups (ie, based on laboratory claims), a subset of national commercial laboratory vendors also included the laboratory result with each claim submitted to the payer. A1C test results were available for about 40% of submitted A1C laboratory claims for both CSII- and MDII-treated patients. Because the availability of an A1C result was determined by where the test was perfomed rather than by a patient’s individual characteristics or form of diabetes treatment, this subsample of individuals with A1C test results enabled a unique opportunity for the analysis of glycemic control within an unbiased subset of patients nationally. The Northwestern University Institutional Review Board reviewed the parent study and determined that this work involved the use of coded, nonidentifiable data and was not classifiable as human subjects research.

Univariate and bivariate descriptive statistics for baseline characteristics were calculated for both CSII initiators and the matched MDII comparison group. Student 

 tests were used to compare continuous variables; χ

 tests were used to compare categorical variables. Means of continuous outcomes were plotted for each 90-day period before and after the index date to observe comparability of time trends between groups.

To minimize the potential for bias introduced by the nonrandomized study design, we used a pre-post with propensity-matched comparison group design. We estimated mean between-group differences in quarterly outcomes over different time horizons (year 1, year 2, year 3) using enrollee-level difference-in-differences random effects regression models that included a dummy variable for group (CSII initiator versus MDII), calendar year indicators (ie, calendar year of the index date), exposure year (ie, year 1, 2, or 3 relative to the index date), and group-by-exposure-year interaction terms. Means and 95% confidence intervals (CIs) were estimated using a generalized method of moments approach. For estimating continuous outcomes, we used linear models because they provide estimates in natural units of the outcome variable and have been shown to produce reliable and unbiased estimates of mean cost differences and CIs when sample sizes are large.

 For count outcomes (eg, hypoglycemic encounters or ED visits), estimations used negative binomial distributional assumptions. Analyses were conducted on the overall sample as well as across prespecified patient subgroups, including different pump-qualifying diagnoses—hypoglycemia encounters or evidence of poor glycemic control within 180 days of the index date&mdash;and whether or not the CSII initiator also used real-time continuous glucose monitoring (rt-CGM) (see eAppendix Table 1).

Because our analysis assumed a health payer perspective, each patient who was no longer enrolled with the health plan was censored from future measurement periods, rather than imputing missing data or assuming them to have zero costs. To avoid potential bias from differential dropout rates across the 2 groups, we censored both the CSII initiator and matched control patient at any point when either one was no longer enrolled in the health plan. This reduced sample sizes but ensured comparability throughout the evaluation period.

The baseline characteristics of 2539 CSII initiators and 2539 matched controls are presented in Table 1. There were no statistically significant differences in any major characteristics between the 2 comparison groups, with good balance on sex, age, comorbidity index score, frequencies of encounters for hypoglycemia or hyperglycemia, and visits to endocrinologists. About half of patients in both groups were women and almost two-thirds were 40 years or older. Only 2.1% of controls and 2.8% of CSII initiators had a hypoglycemia billing code in the 90 days prior to the index date. More than one-fourth of CSII initiators did not have an encounter with an endocrinologist in the 6 months before the index date, suggesting management by another type of provider. For the subgroup of patients who used laboratory vendors that provided test results to the payer (about 31% of patients in each treatment group at baseline), about 60% had baseline A1C concentrations greater than or equal to 8%.

Among CSII initiators, 25% were no longer enrolled after 24 months. For those who remained enrolled, 8.5% had resumed regular refills of basal insulin and had no further evidence of pump-related supplies by the end of year 1; 13.9% had assumed this pattern by the end of year 2. There were no statistically significant differences in mean age, sex, or comorbidity scores between patients who remained with their health plan and those who did not remain enrolled. Among those who remained enrolled, the probability of stopping CSII use was higher among patients younger than 50 years compared with those aged 50 to 64 years (15.3% vs 11.6%; 

Figure 2 (panels A-D) displays the mean per-person-per-quarter healthcare expenditures of CSII initiators and matched MDII users before and after index dates. Cost trends between groups were comparable before the index date, but there was a small uptick in total expenditures for the pump users during the first 6 months after the index date (ie, during the pump “onboarding” period). This increase was associated with claims for pump-related services, such as education and laboratory testing required for pump authorization, followed by a subsequent large spike in total medical expenditures during the first quarter (Q) after receiving the pump (Q3 for most clients), which related to the pump device (mean cost = $4786) plus supplies, additional educational encounters, and follow-up visits. Modest differences in ambulatory medical and pharmacy expenditures also persisted after Q3, reflecting ongoing expenditures for pump supplies and follow-up encounters.

Predicted differences in mean healthcare expenditures per person per quarter are displayed in Table 2. Mean per-person total healthcare expenditures during year 1 were estimated to be $2247 higher per quarter for CSII initiators than for matched MDII patients, amounting to a total difference of about $9000 over the first year. Over the entire 3-year evaluation period, the adjusted mean per-person total healthcare expenditures were estimated to be $1714 (95% CI, $1184-$2244) higher per quarter for CSII initiators compared with matched MDII patients, amounting to a total mean difference of about $20,565 over 3 years. There were no statistically significant differences during any follow-up year in potentially offsetting utilization, such as ED visit rates, inpatient utilization, or ambulatory expenditures that were not directly related to CSII therapy (data not shown).

Glycemic Control and Hypoglycemia Encounters

For the subgroup analysis of 31% of clients with available A1C results, trends in mean A1C concentrations prior to the index date were well balanced, but then increased slightly for CSII initiators during the pump onboarding period (Q1-Q3) (data not shown) before dropping precipitously near the time of pump initiation. The mean A1C was estimated to be 0.46% lower for CSII initiators than MDII controls in year 2 (

= .047). However, CSII initiators also had a higher rate of hypoglycemia encounters throughout year 1 after their index date (

Comparative Effectiveness Among Prespecified Patient Subgroups

Figure 3 displays quarterly trends in total healthcare expenditures for the prespecified subgroups of CSII initiators: presence/absence of encounters for hypoglycemia (panels A and B), presence/absence of encounters for uncontrolled or labile blood glucose (panels C and D), and concomitant use/nonuse of rt-CGM in the first 2 years of CSII initiation (panels E and F). These comparisons show that CSII initiators with either uncontrolled blood glucose or hypoglycemia encounters tended to have higher mean quarterly healthcare expenditures both before and after the index date than did CSII initiators without those qualifying conditions. In addition, the expenditure trends following CSII initiation did not seem to differ significantly between those qualifying conditions. Third, concomitant use of rt-CGM (39.6% of CSII users) appeared to have no measurable impact on CSII expenditure trends.

This natural experiment provides new information that adults with insulin-requiring diabetes who transition from MDII to CSII have mean total healthcare expenditures over a 3-year period that are about $1700 higher each quarter, equating to about $20,500 more per person over 3 years compared with otherwise similar patients who continue to use MDII. Although we did observe a favorable mean difference in A1C of 0.46% in year 2 and 0.32% in year 3, we also observed statistically significant differences in hypoglycemic encounters, which may indicate an adverse effect of intensification in some patients or a need for additional education or training. Although we explored if there might be different expenditure trends for prespecified patient subgroups, we observed no differences between patients with or without recent encounters for hypoglycemia, with or without evidence of poor glucose control, or those who used rt-CGM concomitantly with CSII.

Cost differences were driven primarily by 3 types of pump-related expenditures: 1) the infusion pump, 2) pump insulin and other supplies, and 3) the incremental use of health services required for ongoing pump coverage. Those costs began over a median of 145 days prior to initiation of the pump and continued after CSII initiation throughout the entire 3-year analytic time horizon. Because patient-level costs and utilization were matched before the index date, incremental ambulatory expenditures were unlikely to be caused by differences in diabetes severity between the 2 comparison groups. Conversely, these expenditures may have resulted from coverage requirements that CSII patients meet certain indications (eg, hypoglycemia, poor glucose control) or complete certain services (diabetes self-management training quarterly physician visits) in order to retain pump coverage. To the extent that patients pay cost sharing amounts for those services, these incremental costs are likely to impact not only the payer but also patients and healthcare providers.

First, we only had access to administrative data sources, which do not capture some meaningful outcomes, such as reduced pain and distress, improved QOL, or mild to moderate hypoglycemic events. If these unmeasured outcomes improve significantly for select patients who are offered CSII, it may alter conclusions about the balance of value and costs. Second, adherence to pump therapy could not be verified. We censored individuals from the analysis when they appeared to have stopped refilling pump supplies and insulin while resuming regular refills of basal insulin, but we had no way of verifying whether CSII users continued to use the pump as prescribed. Third, we were only able to analyze direct medical costs. Patients may also incur nonmedical costs (eg, transportation to doctor visits) or indirect costs (eg, missed workdays) after receiving different therapies or services. Last, although longer than most randomized controlled trials, the analysis period in our study of 3 years is still relatively short. In those for whom CSII improves glycemic control, it may require a longer period of time for this beneficial outcome to translate into lower overall healthcare utilization and costs.

Considering the ongoing movement toward value-based payment and delivery systems, there will be increasing incentives to offer management approaches that have higher upfront costs only when there is a strong likelihood of benefit to the patient. One important implication of our findings is that it may remain challenging for health payers or providers to develop policies regarding access or coverage for CSII when, in aggregate, CSII appears to add very high immediate costs with uncertain and/or difficult-to-measure short-term benefits. Unfortunately, our analysis did not uncover a clear patient subgroup for which the comparative effectiveness or costs of transitioning to CSII were more favorable in the short term. Payers already require a relatively strict set of criteria to be documented before approving CSII. These steps are intended to ensure CSII is appropriate and that the patient is fully informed and motivated. However, these efforts also have an unintended effect of increasing healthcare utilization and costs even before an insulin pump is dispensed, with some patients never actually initiating CSII and others reverting to MDII within only a few months. For CSII to be viewed as high value, additional research will be needed to inform more efficient strategies that identify patients who are most likely to benefit, as well as to provide the right balance of clinical education and support to prepare those patients for CSII and maximize its potential benefit.

 Department of Medicine, Northwestern University Feinberg School of Medicine (RTA, AW, RK, AC), Chicago, IL; UnitedHealth Center for Health Reform and Modernization, UnitedHealth Group (TAP, LGS, DV), Minnetonka, MN.

 Dr Ackermann, Mr Kang, and Mr Cooper were supported by grant U58DP002718 from the Centers for Disease Control and Prevention and National Institute for Diabetes and Digestive and Kidney Diseases as part of the Natural Experiments for Translation in Diabetes (NEXT-D) Study.

 Drs Sandy and Vojta and Mr Prospect are employees and stockholders of UnitedHealth Group. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (RTA, AW, RK, TAP, LGS, DV); acquisition of data (AC, TAP, DV); analysis and interpretation of data (RTA, AW, RK, AC, TAP, LGS, DV); drafting of the manuscript (RTA, RK); critical revision of the manuscript for important intellectual content (AW, AC, TAP, LGS, DV); statistical analysis (RK, AC); provision of patients or study materials (TAP, DV); obtaining funding (RTA); administrative, technical, or logistic support (AC).

 Ronald T. Ackermann, MD, MPH, Northwestern University School of Medicine, 750 N Lake Shore Dr, Ste 680, Chicago, IL 60611. E-mail: r.ackermann@northwestern.edu. 

1. National diabetes statistics report: estimates of diabetes and its burden in the United States, 2014. CDC website. https://www.cdc.gov/diabetes/pubs/statsreport14/national-diabetes-report-web.pdf. Published October 24, 2014. Accessed January 12, 2016.

2. Nathan DM, Genuth S, Lachin J, et al; Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. 

3. Brown SE, Meltzer DO, Chin MH, Huang ES. Perceptions of quality-of-life effects of treatments for diabetes mellitus in vulnerable and nonvulnerable older patients. 

. 2008;56(7):1183-1190. doi: 10.1111/j.1532-5415.2008.01757.x.

4. Bloomgarden ZT. Aspects of insulin treatment. 

. 2010;33(1):e1-e6. doi: 10.2337/dc10-zb01.

5. Reznik Y, Cohen O. Insulin pump for type 2 diabetes: use and misuse of continuous subcutaneous insulin infusion in type 2 diabetes. 

. 2013;36(suppl 2):S219-S225. doi: 10.2337/dcS13-2027.

6. Standards of medical care in diabetes—2015: summary of revisions. 

. 2015;38(suppl):S4. doi: 10.2337/dc15-S003.

7. Grunberger G, Abelseth JM, Bailey TS, et al. Consensus statement by the American Association of Clinical Endocrinologists/American College of Endocrinology insulin pump management task force. 

. 2014;20(5):463-489. doi: 10.4158/EP14145.PS.

8. Heinemann L, Fleming GA, Petrie JR, Holl RW, Bergenstal RM, Peters AL. Insulin pump risks and benefits: a clinical appraisal of pump safety standards, adverse event reporting and research needs. a joint statement of the European Association for the Study of Diabetes and the American Diabetes Association Diabetes Technology Working Group. 

. 2015;58(5):862-870. doi: 10.1007/s00125-015-3513-z.

. 2007;25(2):41-42. doi: 10.2337/diaclin.25.2.41.

10. Skylar JS, Ponder S, Kruger DF, Matheson D, Parkin CG. Is there a place for insulin pump therapy in your practice? 

11. Yeh HC, Brown TT, Maruthur N, et al. Comparative effectiveness and safety of methods of insulin delivery and glucose monitoring for diabetes mellitus: a systematic review and meta-analysis. 

Articles

CLINICAL

Evaluation of healthcare utilization and costs over 3 years for adults with insulin-requiring diabetes who transition from multiple daily insulin injections to insulin infusion pumps.

Comparative Effectiveness and Costs of Insulin Pump Therapy for Diabetes

Clinical trials are increasingly testing the effectiveness of paying patients’ financial incentives for achieving desired clinical outcomes. Some researchers and providers are concerned that patient financial incentives will harm the doctor—patient relationship. How patients feel about these approaches, and these trials, is largely unknown. This study examined patients’ perceptions of a compound behavioral and financial incentive intervention used in a large multicenter trial to lower low-density lipoprotein cholesterol (LDL-C), including their perceptions of benefits and challenges and the study’s effect on patients’ relationship with their primary care physicians (PCPs). 

Semi-structured telephone interviews with patients post intervention. 

PCPs from 3 primary care practices in the northeastern United States were randomized to 1 of 4 arms: physician financial incentives, patient financial incentives, shared incentives between physicians and patients, and a control arm. Within each arm, 10 high, 10 medium, and 10 low performers in LDL-C reduction were interviewed. Interviews targeted reasons for enrolling in the study, the specific intervention elements that helped them reach the goal (incentives, engagement, monitoring), challenges faced in reducing cholesterol, and the impact of study participation on their relationship with their PCP.

Patients reported positive experiences with the study: 65% described personal changes to improve health and 61% reported increased awareness. Views about financial incentives varied: 71% clearly found them motivating and 36% claimed they made no difference. Patients noted that changing lifestyle (36%) and diet (65%) was difficult. Patients who substantially lowered their LDL-C revealed themes similar to those who did not. 

Overall, behavioral interventions with financial incentives appear to be socially acceptable to patients who participate in them. Both adherence monitoring and financial incentives were well received, with little effect on the physician-patient relationship.

Patients who enrolled in a trial to lower low-density lipoprotein cholesterol (LDL-C)—one that included financial incentives to patients or to physicians or were shared between patients and physicians&mdash;spoke positively of the multifaceted intervention. One-on-one interviews revealed:  

Wireless pillboxes for monitoring served as a reminder to take medications. 

Blood draws every 3 months were reinforcing and informative. 

Financial incentives were appreciated but not viewed as particularly motivating. 

Patients who substantially lowered their LDL-C revealed themes similar to those who did not. 

Financial incentives were rarely discussed; they did not disrupt the patient—physician relationship. 

Overall, financial incentives appear to be effective and socially acceptable.

Patient nonadherence to recommended treatment is a barrier to improving outcomes, but the use of financial incentives to overcome this barrier is increasing.

 Eighty-five percent of large employers use financial incentives to improve health behaviors in realms ranging from smoking and other addictions to weight control and medication adherence.

 In a recent randomized controlled trial focused on reducing low-density lipoprotein cholesterol (LDL-C) among patients with high cardiovascular risk or established cardiovascular disease, incentives based on principles of behavioral economics were used to motivate both patients and providers.

 In the parent study, 340 primary care physicians (PCPs) from 3 health systems and about 15,030 of their patients were enrolled into a 4-arm trial to test alternative incentive strategies to lower LDL-C. PCPs were randomized to 1 of 4 arms: physician financial incentives, patient financial incentives, shared incentives between physicians and patients, and a control arm. Progress was monitored through quarterly LDL checks and medication adherence via pill bottle opening. The primary outcome was LDL reduction over 12 months. The intervention was a compound of multiple behavioral economic strategies.

The main trial showed significant benefits for those in the shared incentive arm, but the effectiveness of the intervention ranged across individuals. The purpose of this current study was to investigate patients’ perceptions of the intervention and learn from their stories of how the intervention affected their behaviors. The specific objectives were to investigate patients’ perceptions of financial incentives and of multiple components of the behavioral intervention, including overall thoughts about benefits and challenges and how being in the study affected their relationship with their physician. We were particularly interested in how perspectives differed between patients who succeeded in lowering their LDL-C and those who did not. Secondarily, we examined if narratives about incentives differed among patients randomized to different arms of the study.

Overview/Summary of the Underlying Intervention

The protocol for the main trial is summarized briefly here.

 A total of 1503 patients with high cardiovascular risk or established cardiovascular disease were enrolled; they were nested within 340 PCPs from the University of Pennsylvania (Penn), Geisinger Health System (GHS), and Harvard Vanguard Medical Associates (HVMA). The study compared 4 approaches to reduce LDL-C: physician incentives, patient incentives, shared physician—patient incentives, or control (no financial incentive or regular feedback). For the patient and physician incentive arms, the maximum expected value of the incentive was $1024 per enrolled patient who met LDL-C goals. PCPs and patients in the shared arm shared those possible financial incentives. Patients also received surveys, quarterly LDL-C checks, and wireless pill bottles (Vitality GlowCaps) which were used to store their cholesterol-lowering medication and electronically monitor bottle openings.

In this study, semi-structured telephone interviews were conducted with patients post intervention from February 2014 through July 2014. The methods and script were approved by the University of Pennsylvania Institutional Review Board.

Purposive targeted sampling was used. Within each of the 4 arms, the goal was to interview 10 high, 10 medium, and 10 low performers in LDL-C reduction, stratified across the 3 study sites. Within each of the 12 cells, 5 patients were targeted from GHS, 4 from Penn, and 1 from HVMA, proportional to the total patient sample at each site. Patients were interviewed in multiple waves, formed as groups of patients completed their 15-month visits. The final number of interviews is shown in 

. For each wave of completed data, usually for 60 to 80 patients, the mean change in LDL-C (baseline to 12-month) was calculated, then sorted by site within arms. Within each arm, the distribution was inspected and the 4 to 5 patients with the most improved, most worsened, and least changed LDL-C were prioritized, choosing only 1 patient per clinical site per wave. Therefore, “high” and “low” do not have a constant cut-off. However, across the selection waves, median changes in LDL-C among targeted patients were similar.

The interview script was developed by a study team of health economists, physicians, and qualitative researchers. They drew on the literature and their expertise to develop conversational questions related to the study elements and the potential impact of behavioral interventions. Initial scripts were piloted and revised as necessary prior to the study and were constant for the duration of active interviewing. Revisions were based on queries about unclear questions, and the order of the questions was changed to improve flow. Pilot interviews are not included in the analyses. Key questions targeted reasons for enrolling in the study, parts of the study that helped them reach the goal (surveys, wireless pill bottle, cholesterol check every 3 months, and incentives, for patients in arms that received them), challenges faced in reducing cholesterol, and the effect of the study on their relationship with their doctor.

Two trained interviewers conducted the interviews. Recordings of the interviews were transcribed by an independent transcription agency. De-identified transcripts were imported into NVivo version 10 (QSR International Pty Ltd) for coding and analysis.

Four trained coders analyzed the data using elements from grounded theory. The team first identified broad themes based on the first 6 transcripts and then applied open codes to the transcripts. The initial set of codes was tested on each subsequent interview transcript to refine the codebook. Codes were revised iteratively by constant comparison between and within transcripts, and more content-driven axial codes were applied. A final coding scheme was established and applied to all transcripts. In a post hoc analysis, the comments about financial incentives were grouped by arm, blinded, and reviewed by the senior author, with a goal of understanding the main trial results. Inter-rater reliability was assessed on samples exceeding 20% of transcripts. The research team met weekly to resolve discrepancies via consensus.

In the following section, results are organized around the major research questions, examining perceived benefits and facilitators of study enrollment and components, barriers to study participation and meeting goals, and how study participation affected their relationships with their doctors. Results focus on themes mentioned by more than 20% of the sample. The median kappa was 0.92, with a range of 0.64 to 1.0.

A total of 110 of 120 planned interviews were completed (Table 1). They came from a targeted list of 163 patients, representing a minimum 67.5% response rate; 9 patients refused and the remainder were not reached prior to saturation. Patients providing interviews were demographically and clinically similar to those in the parent trial (

Prevalent themes are presented in italics, with quotations summarized in the 

). There were few differences between patients whose LDL-C improved greatly and those who did not improve. Similarly, meaningful differences among study arms were not prevalent. When substantial differences were discovered, they are noted at the end of each section.

Patients expressed a range of reasons for joining the study. First, the 

 of a doctor’s recommendation to join the study was a motivating factor for 40 (36%) patients. Second, 34 (31%) patients stated that they joined the study with the 

 to their high cholesterol. A third reason for joining was 

n = 30; 27%). Other relatively common motivators were 

(n = 27; 25%), including free and more frequent monitoring, and the 

 of helping with research (n = 25; 23%). The subgroup of interviewees whose LDL-C worsened gave more responses to this question, but the nature of what they said was not qualitatively different. The subgroup whose LDL-C improved often mentioned enrolling to facilitate healthy behaviors and to learn more about cholesterol issues.

Patients offered a wide range of responses to queries about the benefits of enrolling in the study. Seventy-one (65%) patients described 

 that affected the way they manage their health and their understanding of health-related topics. Commonly referenced topics were diet choice and exercise regimens. The study made patients think more about the foods that they were eating and the necessity of changing eating habits. These changes included reading labels before buying food, shopping for healthier food options, and portion control. The study also inspired patients to engage in more 

. Some learned that they were able to control their cholesterol with diet and exercise and others learned the necessity of medication in controlling their cholesterol.

Just over half (n = 61; 55%) of the patients spoke of appreciating 

. Several patients explained that being in the study made them more aware of the severity of their health condition; 34 (31%) specifically mentioned improved health. The subgroup of interviewees whose LDL-C got better more often gave examples of how they changed a health behavior and more frequently spoke of more consistent use of medication.

Benefits and Facilitators: Intervention Components

 More than half (n = 69; 63%) of the patients mentioned the benefits of receiving blood test results every 3 months; they 

, and tracking enabled them to make appropriate adjustments as well as understand how medication use affected their cholesterol levels. It was 

 to see how diet and exercise changes affected their cholesterol. They also appreciated the increased access to services (n = 48; 44%) and did not find the visits inconvenient (n = 47; 43%)

 Patients generally agreed that the wireless devices were a facilitator in lowering their cholesterol. The devices served as a 

to take medication (n = 39; 35%). Many thought the devices were easy to use and expected to continue to use them after the conclusion of the study. They liked being monitored (n = 33; 30%)

 Patients did not say much about the surveys. The most prevalent comment was about 

(n = 40; 36%). The surveys provided a friendly reminder of the appropriate steps that patients should be taking to keep their cholesterol under control.

 Seventy-five patients provided comments about financial incentives. Many (n = 53; 71%) described the 

 that the incentives had on their behavior, offering at least 4 different examples: 1) several thought it was good that the study provided them with 

 and have some extra money to spend on other things, 3) multiple patients stated that the incentives made the other inconvenient aspects of the study more 

, and 4) several patients felt the positive reinforcement 

 them to do the necessary things to lower their cholesterol and continue with the study. Most of the 42 (56%) who commented felt that the 

At the same time, 27 (38%) patients said that the incentives 

 on their performance in the study; positive results were what motivated them to keep up their efforts. They were involved in the study to get healthier and learn about themselves. The blinded review of the comments about incentives yielded nothing to distinguish among the treatment arms with patient incentives.

Only 11 (15%) patients recalled talking about the incentives with their doctors; these conversations were brief affirmations of their participation in the program and encouragement to continue with their progress. Overall, 38 (51%) patients made some commentary about physicians and incentives. Although there was no prevalent theme, the tone was generally indifferent (eg, it would not affect the patient’s own performance) or supportive (eg, they were happy for their doctors to receive payment as part of the study). Few feared that physician incentives would affect the physician’s impartiality.

There were no differences among subgroups of patients related to discussions about the 3-month visits, wireless medication devices, or the surveys. The subgroup of interviewees whose LDL-C improved more often spoke about the benefits of financial incentives.

 In contrast to the many experienced benefits, some barriers to improving cholesterol remained. The main downfall of 72 (65%) patients was their inability to stay away from food that is believed to contribute to higher LDL-C levels, and for others (n = 40; 36%), it was their inability to maintain a healthy lifestyle. A host of reasons were given, including

 who regularly ate or prepared unhealthful meals, 

. Comments about inadequate activity were often mentioned in the same context as 

. Multiple reasons were provided, often related to the weather, amount of time required for work, travel, or 

. Issues regarding the difficulties of maintaining a healthful diet and lifestyle were more prevalent among the subgroup whose LDL-C worsened over the study period.

 For more than half (n = 67; 61%) of the patients and for a number of different reasons, the wireless medication device used in this study (Vitality GlowCap) was seen as a nuisance, but usually temporarily. Patients on multiple medications did not like the necessity of having 

such as finding an appropriate outlet and connectivity difficulties; but for most, these were transitory issues amid an overall positive experience. The subgroup that did particularly well in improving its LDL-C more often noted issues with the wireless medication devices, but also that the issues were infrequent and/or solved.

 Numerous patients (n = 49; 45%) referenced regular medication use as a challenge. A significant part of the commentary was about 

, such as muscle weakness and pain and liver complications. Side effects, such as feelings of illness, muscle and joint pain, and “fuzzy memory,” were cited as reasons for dropping the medication altogether. Other patients 

 to take their medicine on a daily basis, and a significant number of patients stated they were 

 in taking any medication at all. The frequency and content of comments was similar between the high and low performers.

Only 1 patient specifically noted a change in the relationship between the patient and physician. However, 29 (26%) indicated that the study led to more conversations with their doctor—typically about the 

, especially if the labs demonstrated “poor results.” Some noted that they saw their physician more often due to participating in the study.

enrolled in the study because their doctor suggested

 it to them. Many of these were from the group who did well. Most of these comments noted that a brief conversation took place during the course of an office visit, in which the doctor brought up the study and recommended participation, and the patients subsequently agreed to join.

This qualitative process evaluation of a large randomized trial has 4 main findings. First, patients reported positive experiences with the study; they enrolled based on a genuine desire for a healthier life. The structure of the study was similarly favorable: blood draws every 3 months were informative, and the wireless medication devices—although they not without their glitches&mdash;provided valuable reminders to take one’s medication. Second, views about financial incentives were varied. Some clearly found them motivating, whereas others claimed they made no difference. Importantly, no patients found them negative. In general, discussion of the study with their physicians, including financial incentives, was not extensive. Third, patients noted that changing behavior and diet was difficult. Food preferences, habits, and social times interrupted plans to make positive changes. There was also ambivalence about taking medication, and for some, there was a clear desire to avoid medication. Fourth, patients who substantially lowered their LDL-C revealed themes similar to those who did not.

These findings contrast with some of the concerns the study team had when developing the study, as well as those voiced by others.

 In preparation for study launch, some team members and participating physicians wondered whether explicit financial incentives might adversely affect existing physician—patient relationships. Those concerns seem to have been unjustified. There was also concern that the wireless medication adherence devices would be seen as intrusive, but they were not. Most importantly, different experiences and feelings based on different intervention arms were anticipated; it seemed logical to think that whether a patient or their physician is being paid for outcomes might alter one’s sense of the process. In fact, no differences were found across arms. Nor were there systematic differences based on how well patients did in the intervention arm in terms of improving their LDL-C.

There is little previous research examining patient perceptions of financial incentives to improve their health. One other published paper found nearly equal proportions of a clinic population had favorable or unfavorable opinions of incentives, but the authors noted greater enthusiasm when incentives were framed as a reward rather than a penalty, when they were offered by health insurers as part of benefit design, and when the survey respondent personally had the condition for which the incentives were offered (in the case of smoking or obesity).

 Importantly, from a complementary perspective, this previous study was conducted among clinic patients who had not been enrolled in a trial of financial incentives.

Patients in the underlying study had to consent to participate; thus, by design, only the experiences of those who were willing to engage in the first place are represented in this study. High and low performers were oversampled, so it is possible that this sample was not effectively representative; however, the observation that no differences were observed between the extremes suggests that no differences would be observed among a more distributed sample. This study also has strengths: the team conducting this evaluation was different from the team that designed and implemented the trial, the sampling was systematic, and the sample included 110 patients, which is substantially larger than many process evaluations.

Behavioral interventions that include financial incentives offer promise to improve important health outcomes, but they must overcome 2 important evidentiary hurdles before widespread implementation. First, they need to demonstrate effectiveness. The results of the underlying study in this report suggest that a system of shared incentives offers significant benefit compared with a control incentive. Second, they must be socially acceptable. There are many stakeholders in such acceptance, including those who might finance these interventions, including the incentives, and the general public who might pass judgment upon them. This study's results suggest that patients are willing to engage in these programs and generally felt participating was a positive experience.&ensp;

The authors would like to acknowledge the team from the Mixed Methods Research Lab at the University of Pennsylvania who participated in conducting interviews and coding transcripts: Shimrit Keddem, Trocon Davis, Ebony Easley, Samuel Katz, Jalal Osman, Breah Paciotti, Ashley Ryman, and Richard Sorelle.

 Division of General Internal Medicine, Department of Medicine (JAS, KGV, DAA), Department of Medical Ethics and Health Policy (KGV, DF, DAA), Department of Biostatistics and Epidemiology (ABT), Center for Health Incentives and Behavioral Economics (KGV, KH, DAA), and Center for Health Care Innovation (KGV, DAA), Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA; Ohio State University Medical School (AA), Columbus, OH; Corporal Michael J. Crescenz 

 (KGV, DAA), Philadelphia, PA; The Wharton School, University of Pennsylvania (KGV, DAA), Philadelphia, PA; Atrius Health (TI), Newton, MA; Harvard T.H. Chan School of Public Health (MR), Boston, MA; Partners Healthcare System (TDS), Boston, MA.

MANAGERIAL

Patients who enrolled in a trial to lower low-density lipoprotein cholesterol spoke positively of the multifaceted intervention: pillbox monitoring and financial incentives were socially acceptable.

Patients' Views of a Behavioral Intervention Including Financial Incentives

 To understand the views of prominent organizations in the field of healthcare quality on the topic of reporting roll-up measures that combine indicators of multiple, often disparate, dimensions of care to consumers.  

This study used a semi-structured, qualitative interview design.

We conducted 30- to 60-minute semi-structured telephone interviews with representatives of 10 organizations that sponsor public healthcare quality reports and 3 national alliances representing multiple stakeholder groups. We conducted a thematic analysis of interview transcriptions to identify common issues and concerns related to reporting roll-up measures.

Among sponsors reporting roll-up measures, current practices for calculating and reporting these measures are diverse. The main perceived benefit of reporting roll-up measures is that they simplify large amounts of complex information for consumers. The main concern is the potential for consumers to misunderstand the measures and what associated roll-up scores communicate about provider performance. Report sponsors and national alliances feel that more guidance and research on the methods for producing and reporting scores for roll-up measures are needed.

The results of the interviews elucidate the need for research focused on construction and reporting of roll-up measures. Studies are needed to determine if roll-up measures are indeed perceived by consumers as being less complex and easier to understand.  

Some organizations that sponsor public reports of healthcare quality present roll-up measures that combine indicators of multiple, often disparate, dimensions of care. We found that: 

Report sponsors’ practices for calculating and reporting these measures are diverse. 

Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers but run the risk of misinterpretation. 

Report sponsors and national alliances indicate a need for more guidance and research on how to produce and report scores for roll-up measures.

A major objective of public reports on healthcare quality is to enable consumers to make well-informed choices about their care. To meet this objective, report sponsors often use strategies intended to reduce the cognitive burden of healthcare quality data and improve consumers’ comprehension and use of the information.

 One strategy is the provision of “roll-up” measures (ie, summary measures of healthcare quality that combine indicators of multiple, often disparate, dimensions of care). For example, a roll-up measure of patient experience with primary care could aggregate composite measures of patient—provider communication, access to care, and interactions with office staff. At a higher level of aggregation, a roll-up measure of overall hospital quality could combine measures of patient experience, clinical processes, and patient outcomes. The most prominent use of roll-up measures within healthcare is CMS’ Five-Star Quality Rating System, which assigns an overall rating of 1 to 5 stars to providers and plans based on multiple quality dimensions.

 Roll-up measures could also aggregate component measures at other levels. 

Publicly reporting roll-up measures offers advantages to consumers insofar as the roll-ups are created scientifically and grounded in high-quality research. These advantages include offering consumers a summary of multiple dimensions of care and, in many cases, a measure of healthcare quality that has superior reliability.

 However, this strategy is not without controversy:  roll-up measures may obscure important nuances of quality and not align with consumer preferences or information needs. Also, the calculation of roll-up scores relies on assumptions about the relative importance of each component measure.

We aimed to understand how prominent organizations in the field of healthcare quality view the reporting of roll-up measures and obtain examples of current approaches to reporting roll-up measures. To that end, this study gathered information from representatives of report sponsors and national healthcare alliances on current practices and positive and negative perceptions with regard to creating and reporting roll-up measures to consumers. We sought to identify these organizations’ concerns given the current interest in reporting these measures to consumers and to determine the need for additional evidence to address these issues.

In this qualitative study, we conducted 30- to 60-minute semi-structured telephone interviews with representatives from 10 organizations that publish comparative reports on healthcare quality (Table 1) and 3 national alliances that bring together multiple stakeholder groups. These alliances—the AQA (formerly the Ambulatory Care Quality Alliance), Consumer-Purchaser Alliance, and the National Quality Forum&mdash;were strategically selected for their expertise in healthcare quality measurement and reporting.

Interviews with report sponsors explored their perceptions of the benefits and drawbacks of reporting roll-up measures, the factors that influenced whether they report these measures, and for those that do report, how they calculate and report these measures to consumers. Interviews with the national alliances focused on whether the reporting of roll-up measures had been discussed within the organization, perceptions of the most important issues or concerns, the types of stakeholders involved in the discussion, and whether they created position statements or recommendations on reporting roll-up measures. Our qualitative approach enabled participants to describe their perceptions and experiences in their own words and was well-suited to eliciting participants’ diverse perspectives.

We used thematic analysis to extract concepts that emerged from multiple interviews.

 Each interview transcription was reviewed by the first author (who also conducted several interviews with report sponsors and all 3 interviews with national alliances). The author extracted key information related to roll-up measure construction and reporting, identified consistent themes across the interviews, and produced short memos describing themes and the evidence for each. After clarifying assumptions and reviewing the themes with the other authors, the author updated the memos to reflect a consensus around the main themes extracted from the data. Key themes were confirmed through an independent review of transcriptions by the fourth author, who did not conduct any of the interviews.

Diversity of Roll-up Measure Reporting Practices

Among sponsors reporting roll-up measures at the time of the interviews, practices for calculating and reporting roll-up measures were diverse. Reporting practices for different providers varied in terms of construction and labeling (Table 2).

 For example, the Wisconsin Department of Employee Trust Funds differentially weighted and combined multiple Consumer Assessment of Healthcare Providers and Systems (CAHPS) and Healthcare Effectiveness Data and Information Set (HEDIS) indicators into a roll-up measure called the “Overall Quality Score” for health plans. In contrast, the Pacific Business Group on Health used an equal-weighting approach to combine several HEDIS measures of medical group performance into the measure, “Medical Group Provides Recommended Care.” 

In the absence of well-established methodological criteria, report sponsors based their decisions about how to calculate and label roll-up scores on information from a variety of sources, including research literature on reporting practices, technical expert guidance, information about other report sponsors’ practices, and the results of psychometric analyses. It is unclear exactly how they incorporated these sources of information into their choices about roll-up measure construction. Some participants reported testing measure labels with consumers or working with advisory groups to construct roll-up measures. All advisory groups included a diverse mix of stakeholders, such as purchasers, providers, state medical associations, and patient advocates. 

The main perceived benefit of reporting roll-up measures is the simplification of large amounts of complex information for consumers. Nearly all participants pointed out that roll-up measures make it easier for consumers to understand and use the information provided in quality reports. For example, one report sponsor representative said, “It’s a huge win for consumers to be able to go in and not have to read through 100 technical measures about hospitals. Even in plain language, 100 measures is a lot of stuff to expect people to go through.” Representatives of the national alliances suggested multiple ways in which reporting roll-up measures could reduce complexity for consumers. One representative asserted that processing roll-up measures requires less effort on the part of the consumer; another argued that reporting these measures allows consumers to gain a comprehensive view of care. In addition to reducing complexity, a third representative suggested that the reporting of roll-up measures could draw attention to the availability of comparative quality information, saying, “…to a large extent, the complexity of existing public reporting has meant that people don’t pay any attention and still ask the neighbors next door, ‘Who do you like?’…so you have to get it to the level that garners attention….” 

However, participants recognized that consumers vary in their need for information and some may want simplified information, but others desire detail (eg, the ability to drill down to scores for the measures that compose the roll-up measures). According to one national alliance representative, high-profile consumer groups argue that when consumers are shown roll-up measures, they should also have the option to view all component measures.

The main concern about reporting roll-up measures is the potential for consumers to misunderstand the measures and what they communicate about provider quality. As one participant noted, if the weighting of individual scores in roll-up measures does not correspond to the relative importance of different aspects of care to consumers, consumers may draw incorrect conclusions about provider performance. Another participant reported that his organization opted against reporting a particular roll-up measure because of uncertainty about how to appropriately weight the component measures. Several report sponsors shared the concern about a potential mismatch between roll-up measures and the salience of individual measures to consumers. One sponsor suggested giving consumers a way to roll up information according to their preferences via an interactive online report.

Several other participants expressed concerns about consumers’ potential misunderstanding of roll-up measures and what they communicate. For example, consumers “might not read the fine print” and, therefore, might miss information about how the roll-up measures were calculated. They may also misunderstand a roll-up measure labeled as “overall quality,” as such a label may imply that all of the important aspects of quality have been measured and are thus represented by the score. In addition, these measures may obscure areas in which providers perform especially well or especially poorly. One report sponsor noted, “That’s always a risk—a person walking away with a generalized sense of performance, but it doesn’t encapsulate the variation of performance of the measures within the composite.”

More guidance and research on the methods for producing and reporting scores for roll-up measures are needed. Several participants expressed a need for more evidence-based guidance on the construction and use of roll-up measures. In the words of one national alliance representative: “I think we could use more science of what logically hangs together, whether there are some things that should never be broken apart, whether there are some things that logically should always hang together.”

We conducted interviews with representatives of report sponsors and national alliances to document attitudes and practices around the calculation and reporting of roll-up measures to healthcare consumers. We identified diverse practices in the construction of roll-up scores; however, our study is limited by its small sample of strategically selected organizations, and we cannot determine the prevalence of roll-up score reporting. The interviews revealed a desire for more guidance and research on the methods for producing and reporting scores for roll-up measures. These findings point to the need for a research agenda focused on developing standards for the construction, calculation, and reporting of roll-up measures that are rooted in empirical data on how constructs assessed by various performance measures relate to each other and desired health outcomes.

One key topic for exploration is the appropriate level of aggregation for roll-up measures. Roll-ups could aggregate measures from a single source (eg, the CAHPS Clinician & Group Survey) or a single domain (eg, clinical quality, patient experience). Alternatively, a single overall roll-up measure, such as the star ratings provided through the CMS Five-Star Rating System, could encompass multiple domains of quality. Although this type of measure may maximize simplicity for consumers, constructing such a score requires strong and nontrivial assumptions about how best to reliably and accurately use a single indicator to communicate the multidimensional construct of healthcare quality.

Although participants perceived that reporting roll-up scores to consumers helps simplify the complex array of healthcare quality information presented in reports, they were concerned about the potential for consumers to misunderstand the aspects of quality captured by roll-up measures and how providers perform on discrete aspects of quality. Whether these beliefs are well-founded is a question worth investigating. Some report sponsors conduct research on roll-up measure construction,

 but the body of published research on consumer use of, and responses to, roll-up measures is minimal.

Consumers are used to seeing roll-ups in other realms (eg, 

 product reviews), but reporting roll-up measures of healthcare quality is relatively new. It is unclear whether consumers find these measures to be easier to understand than the component measures from which they are constructed or how these measures are interpreted. It is also unclear how viewing roll-up measures affects consumers’ choices of plans and providers, what consumers interpret rolled-up measures to signify, or whether roll-up measures simplify available data to the point where differences in performance no longer seem meaningful. Finally, little is known about whether roll-ups might deter crucial forms of consumer learning about performance metrics and provider variation in quality.

Consumer interpretation and use of roll-up measures likely vary depending on the level of aggregation of scores. The benefits and drawbacks of other stakeholders’ use of roll-up measures (eg, providers’ potential use of roll-ups for quality improvement purposes) were not documented here given the focus on report sponsors’ expectations about consumers’ use of roll-up scores; however, these expectations are important to understand. Because the reporting of roll-up measures is underway at federal agencies and among some report sponsors, it is critical to better understand how consumers perceive and respond to such measures. 

We found that report sponsors’ practices for calculating and reporting these measures are diverse. Report sponsors and national alliances that represent stakeholder groups affected by quality reporting feel that roll-up measures simplify large amounts of complex information for consumers, but run the risk of misinterpretation. The results of the study elucidate the need for research focused on the construction and reporting of roll-up measures, as well as consumer interpretations of roll-up measures.

 RAND Corporation (JLC, SCM, MLF, AMP, GM), Pittsburgh, PA; Severyn Group (LR), Ashburn, VA; University of Wisconsin (RG), Madison, WI; Yale University (MS), New Haven, CT; Shaller Consulting Group (DS), Stillwater, MN.

This study identifies practices and perceptions around public reporting of â€œroll-upâ€

June 2017

Comparative Effectiveness and Costs of Insulin Pump Therapy for Diabetes

Patients' Views of a Behavioral Intervention Including Financial Incentives

Using "Roll-up" Measures in Healthcare Quality Reports: Perspectives of Report Sponsors and National Alliances

Does the Offer of Free Prescriptions Increase Generic Prescribing?

Patients' Views on Price Shopping and Price Transparency

Patient Ratings of Veterans Affairs and Affiliated Hospitals

Heterogeneity of Nonadherent Buprenorphine Patients: Subgroup Characteristics and Outcomes

How Do Medicare Advantage Beneficiary Payments Vary With Tenure?

A Comparison of Retrospective Attribution Rules

Radical Prostatectomy Innovation and Outcomes at Military and Civilian Institutions