Patient-centered healthcare is a high priority and is commonly measured and incentivized through patient satisfaction surveys. There is a need to further understand if increasing satisfaction has the unintended consequence of encouraging low-value care. This study assessed the association of low-value antibiotic prescribing with patient satisfaction scores, and it evaluated patient and provider characteristics that may impact the association. 

 Retrospective, observational study of acute sinusitis (AS) encounters for adult members of a large integrated delivery system from 2010 to 2013.

 Bivariate and multivariate analyses evaluating the use of antibiotics, patient attributes, and provider characteristics associated with favorable patient satisfaction scores. 

 Among 5169 encounters for AS, 79.5% of encounters in which antibiotics were prescribed had favorable satisfaction scores versus 75.4% of encounters in which they were not. Independent predictors of favorable satisfaction scores included: receipt of antibiotics (adjusted odds ratio [aOR], 1.24; 95% CI, 1.00-1.55), 45 years or older (aOR, 1.45; 95% CI, 1.24-1.69), Elixhauser Comorbidity Index score 2 or greater (aOR, 1.21; 95% CI, 1.05-1.40), provider business partner status (aOR, 1.38; 95% CI, 1.20-1.58), and a bonded encounter between a patient and their assigned primary care physician (aOR, 2.06; 95% CI, 1.75-2.44).

 Patient satisfaction scores are slightly lower when antibiotics are 

prescribed for AS, but 75% of those encounters still received favorable satisfaction scores. Factors such as older patient age, more comorbidities, and an established patient—provider relationship had stronger associations with high patient satisfaction.

We assessed the effect of low-value antibiotic prescribing on patient satisfaction scores, as well as the impact of patient and provider characteristics on favorable scores. 

Patient satisfaction is slightly higher when low-value antibiotics are prescribed.

A total of 75.4% of encounters not resulting in prescriptions of low-value antibiotics still received favorable satisfaction scores.

Encounter factors such as an established patient-provider relationship, older patient age, and more chronic conditions each favorably influence patient satisfaction scores.

Future research and policies should account for these influential variables when studying or comparing patient satisfaction.

In recent years, patient satisfaction has been recognized as a key measure of the quality of healthcare delivery in the United States. In an effort to encourage a greater focus on patient satisfaction, hospitals and providers have been given financial incentives to measure and improve these scores, especially relative to their peers.

 Although some reports have shown a favorable relationship between high satisfaction scores and other dimensions of quality,

 others suggest that the emphasis on higher scores may promote job dissatisfaction among providers

 or increase potentially harmful low-value care.

 This conflicting evidence makes it difficult to know if efforts to improve patient satisfaction scores facilitate or hinder a health system’s ability to provide high-quality healthcare in other respects.

Another important, and seemingly unrelated issue, is antibiotic stewardship. Prescribing antibiotics for common viral illnesses is widely prevalent but unnecessary, costly, and potentially harmful.

Acute sinusitis (AS), which affects 30 million Americans annually,

 is an example of a common condition for which low-value antibiotics are too often prescribed. US and European studies have all reported high rates (81%-85%) of antibiotic use for AS,

 and in the participating study sites for the current study, the reported prescription rates were as high as 90%. With the majority of these prescriptions being contrary to current recommendations,

 these prescriptions are likely to cause more harm than benefit.

 This low-value prescribing for AS has become such a concern that it is targeted by 5 national provider organizations as part of the Choosing Wisely campaign.

Patient satisfaction is one reason for prescribing low-value antibiotics. Some providers believe that their patients expect to receive prescriptions for antibiotics

 and, in addition, that high patient satisfaction scores depend upon the receipt of such prescriptions.

 Although there has been some research on the relationship between the receipt of antibiotics and patient satisfaction scores, there is insufficient information to draw conclusions about the impact.

 Understanding the patient and provider characteristics that influence the relationship between low-value care and patient satisfaction will inform interventions and policies to promote high-quality care. 

The primary aim of this study was to assess the effect of low-value antibiotic prescribing on patient satisfaction scores. The secondary aims were to evaluate patient and provider characteristics that may confound or modify the association between receipt of low-value care and satisfaction scores. 

This is a retrospective, observational study of AS encounters (

International Classification of Diseases, Ninth Revision, Clinical Modification 

] code 461.x) from 2010 to 2013 within Kaiser Permanente Southern California, a large integrated delivery system that provides care for approximately 4 million members of a prepaid health plan.

The primary inclusion criterion was adults diagnosed with AS. To limit our sample to those with uncomplicated AS, who are least likely to warrant antibiotics, we excluded encounters resulting in hospital admission, follow-up encounters, and encounters for immunocompromised patients. A follow-up AS encounter was defined as a visit for a patient who had already had a visit for AS in the 30 days before the index visit. Follow-up visits were excluded as they are more likely to warrant an antibiotic prescription based on current recommendations.

 “Immunocompromised” was defined as the presence of a diagnostic code for any of the following in the 12 months prior to the encounter: chronic liver disease (

 codes 140-165, 170-176, 179-209, and 235-239), and common rheumatologic disorders frequently treated with immune-suppressing medications (

 codes 714, 710, 555.9, and 556), as described in prior studies.

 The sample for analysis included eligible patients with uncomplicated AS who completed a patient satisfaction survey.

We included the following patient-level characteristics in our analysis: age, sex, race, and Elixhauser Comorbidity Index (ECI) score. The ECI is a validated tool, similar to the Charlson Comorbidity Index, designed to adjust for differences in patient outcomes that are explained by variations among chronic patient conditions.

 Provider characteristics included age, sex, partner status (associate vs business partner), race, years of experience, and patient—provider bonded status. The patient–provider bonded status variable indicates if the encounter occurred between a patient and their assigned primary care provider. Nonbonded visits comprised many encounter types, including urgent care visits (28%) and emergency department or specialty care visits (7%), but most were with another primary care provider (65%) who may have been covering for the assigned bonded provider. For these reasons, we consider the bonded status a reasonable surrogate marker for an established patient-doctor relationship.

We identified our primary outcome by using results from a routine, randomized, postvisit, member satisfaction survey. The primary outcome for analysis was receipt of an overall patient satisfaction score of 9 or greater (out of a possible 10), and the main independent variable of interest was an antibiotic prescription after an AS encounter. The survey was sent at random to members following approximately 15% of outpatient visits, and it had an overall response rate of 18%. Although this response rate is not as high as recommended in primary survey research, it represents pragmatic, real-world results reflective of the patients who respond to current satisfaction surveys used to evaluate providers and health systems in the United States. The satisfaction outcome we used for our analysis was the combined mean score derived from 10 questions asked that were related to visit satisfaction. Questions were scored on a 10-point scale ranging from very dissatisfied (1) to very satisfied (10). We set the “favorable” mean satisfaction score of 9 or greater as the cut point because in our system, providers receive a small financial incentive added monthly to their base pay for scores of 9 or higher. Beyond financial compensation, mean satisfaction scores are also used for credentialing purposes and to assess a physician’s eligibility to be promoted from an associate physician to a partner in the Southern California Permanente Medical Group.

We used hierarchical logistic regression to examine patient and provider factors that were associated with an overall patient satisfaction score of 9 or greater. This allowed us to account for clustering of patients within a provider’s patient panel. First, we performed model selection to identify patient and provider factors that were associated with a patient satisfaction score of 9 or greater in models that included a variable indicating if antibiotics were prescribed. Variables for which 

values were less than a 0.2 level of significance in the first step were advanced to the model selection procedure. Second, antibiotic receipt was included in all logistic regression models, and other variables were then included individually to assess whether receipt of antibiotics remained associated with favorable satisfaction scores, independent of each variable. Variables were included in the final adjusted model if they met one of the following criteria: 1) the 

 value for antibiotics receipt crossed the .05 level of significance (in either direction) after including it in the model or 2) a statistically significant interaction occurred between the variable and the receipt of antibiotics (Likelihood Ratio Test 

 <.05). A final adjusted model, including receipt of antibiotics and the variables or interactions meeting any of the 3 aforementioned criteria, was used to summarize the adjusted relationship between receipt of antibiotics and patient satisfaction scores. Where necessary, we adjusted for multiple comparisons using a Bonferroni correction.

Patient age was dichotomized as being above or below the 25th percentile (45 years), as comparisons of satisfaction scores revealed that while patients in the 50th and 75th percentile age groups had significantly higher scores than those below the 25th percentile, the former 2 did not score providers significantly differently. For similar reasons, we dichotomized the ECI at the 75th percentile (≥2 comorbidities). 

Additionally, we tested an antibiotics-by-bonded-status interaction to assess whether the effect of prescribing differed for patients evaluated by their primary care provider versus another physician and adjusted the stratum-specific odds ratios (ORs) and 95% CIs as appropriate.

 We also performed a sensitivity analysis to identify factors that appeared to be unbalanced (assessed via crude standardized differences and 

values) between antibiotic groups. In addition to the primary outcome of overall patient satisfaction, we performed adjusted sensitivity analyses to assess individual responses to satisfaction questions specific to individual components of the encounter, such as satisfaction with the provider and assessment of the provider’s abilities. All statistical analyses were performed with SAS version 9.3 (SAS Institute Inc; Cary, North Carolina).

The study sample included 5169 encounters for AS with a completed satisfaction survey from 2010 to 2013 (

). Patients had a mean age of 54.9 years, 68.0% were female, 58.4% were white, and the mean ECI score was 1.6. Among providers, 45.6% were female, 53.7% were partners, and 87.5% had more than 3 years’ postresidency experience. Overall, 79.1% of encounters resulted in a favorable satisfaction score: 79.5% of those receiving antibiotics and 75.4% of those who did not (

Unadjusted results showed that prescribing antibiotics increased the odds of receiving favorable satisfaction scores (OR, 1.27; 95% CI, 1.02-1.58). Other variables predicting higher satisfaction scores were patient 45 years or older (OR, 1.58; 95% CI, 1.36-1.83), ECI score of 2 or greater (OR, 1.36; 95% CI, 1.19-1.57), and provider partner status (OR, 1.41; 95% CI, 1.23-1.61). The largest effect was from bonded encounters in which an established patient—provider relationship existed (OR, 2.06; 95% CI, 1.75-2.44). Patient sex (OR, 0.94; 95% CI, 0.81-1.08), white patient race (OR, 1.096; 95% CI, 0.96-1.26), and provider sex (OR, 0.98; 95% CI, 0.85-1.12) were not associated with higher satisfaction scores (

In the multivariable analysis, receipt of antibiotics narrowly maintained statistical significance (OR, 1.24; 95% CI, 1.00-1.55) and other variables remained significantly correlated with high satisfaction scores: patient 45 years or older (OR, 1.45; 95% CI, 1.24-1.69), ECI score 2 or greater (OR, 1.21; 95% CI, 1.05-1.40), and provider partner status (OR, 1.38; 95% CI, 1.20-1.58). There was a statistically significant interaction between receipt of antibiotics and bonded status (

). Adjusted results—using, as the comparison group, nonbonded encounters in which antibiotics were not prescribed&mdash;showed that bonded encounters in which antibiotics were prescribed had the highest odds of receiving favorable satisfaction scores (OR, 2.33; 95% CI, 1.56-3.47). Nonbonded encounters in which antibiotics were prescribed (OR, 1.15; 95% CI, 0.81-1.63) and bonded encounters in which antibiotics were not prescribed (OR, 1.47; 95% CI, 0.77-2.78) were not significantly different from one another. In a sensitivity analysis stratified by patient—provider bonded status, and adjusting the confidence limits for multiple comparisons, we found no significant antibiotic impact on favorable satisfaction scores for bonded encounters (OR, 1.59; 95% CI, 0.89-2.85) or nonbonded encounters (OR, 1.15; 95% CI, 0.81-1.63). 

In this large observational study, we found that prescribing low-value antibiotics increased the odds of a favorable satisfaction score. However, the vast majority of encounters in which antibiotics were not prescribed still resulted in high satisfaction (75.4%), marking only about a 4 percentage-point difference from those in which antibiotics were prescribed (79.5%). The factor with the largest effect on satisfaction was an established patient—doctor relationship; encounters with older patients and those with more chronic conditions were also more likely to result in favorable satisfaction scores.

These results have particular implications for current policies and practices using satisfaction scores to evaluate system and provider performance. The patient and provider characteristics found to be associated with higher patient satisfaction may help explain the contradictory results of different studies, some linking higher quality to higher satisfaction,

 others showing the correlation of adverse outcomes with higher satisfaction.

 Future research should attempt to adjust for patient age and comorbidities, as well as an established patient—provider relationship. 

Providers should focus on the fact that most encounters not receiving antibiotics still had high overall satisfaction scores, not on the small 4 percentage-point absolute difference (Table 1). We predict most providers would not want to prescribe low-value antibiotics to 25 individuals in order to possibly receive a small benefit in 1 satisfaction score. Our results are also particularly useful in understanding how an established patient—provider relationship may play the largest role in contributing to high patient satisfaction scores. This has implications for acute care providers in urgent care and emergency department settings; they should not expect to have satisfaction results similar to those generated by similar encounters between a patient and their primary care provider. An overemphasis on patient satisfaction scores for providers without an established relationship with patients may exacerbate the provision of low-value care or contribute to provider dissatisfaction with existing measures of patient satisfaction. This evidence suggests a need to consider varying patient demographics and variations in established patient–provider relationships to appropriately compare satisfaction scores among different clinical settings.

We recognize the limitations of a retrospective observational study and the possibility that unmeasured confounders might have influenced patient satisfaction. However, we made every effort to control for measurable patient and provider factors influencing satisfaction, beyond those published in previous studies. Also, the physicians in our study are all part of a large prepaid multispecialty medical group and integrated delivery system; therefore, the generalizability of these results may be limited. We were able to identify factors associated with patient satisfaction, but cannot determine if these factors are causal or the putative mechanism of causality. There may also be other potentially important patient factors (eg, socioeconomic status or mental health disorders) that influence patient satisfaction, which we could not measure. Nonresponder bias in our surveys is a consideration, and it is possible that patients seen during a bonded patient—provider encounter might have been more or less likely to complete the survey. However, the survey methods used in our system are similar to those used by others per Medicare requirements. Our study sample of encounters returning satisfaction surveys had similar patient characteristics to our previously published study of all initial AS encounters,

 and our results are representative of current, real-world satisfaction survey reports, which is a more important indicator of the results US physicians and health systems should expect. Also, all surveys are strictly confidential; consequently, patients should not have been concerned that their responses would be seen by their physicians. Additionally, all our encounters use electronic health record documentation and although this has been reported to affect patient satisfaction scores in other settings, a previous analysis showed that using computers in the exam rooms did not impact patient satisfaction scores in our system.

An important strength of this large observational study is that we were able to examine satisfaction scores at the encounter level, specifically with respect to whether a patient received antibiotics or not. This contrasts with many prior studies in which the satisfaction of patients was studied at an aggregate level.

We hope that our findings will not encourage unwarranted antibiotic prescribing, or any other low-value medical services, in order to bolster patient satisfaction scores. Our results are intended to inform future research and policies, and they are similar to recent facility-level comparisons in the United Kingdom, which concluded that patients are more satisfied when they receive antibiotics.

 These findings highlight the importance of measuring, reporting, and incentivizing providers based on quality metrics beyond patient satisfaction. However, the absolute effect is small and achieving high patient satisfaction is a complex issue that should not trump sound medical care and quality.

Our findings indicate that patient satisfaction scores are slightly lower when antibiotics are not prescribed for some encounters. However, the minimal overall impact may help to assure providers that prescribing unwarranted antibiotics is not required to receive high satisfaction scores, as most (75.4%) AS encounters not receiving antibiotics still resulted in favorable satisfaction scores (vs 79.5%). Additionally, an established patient—provider relationship and encounters with older patients with more comorbidities are more likely to be correlated with high satisfaction scores for acute encounters. These characteristics should be included in future satisfaction research and hospital/provider comparisons. The correlation between low-value care and higher satisfaction should encourage policies to evaluate and incentivize providers not only for high patient satisfaction, but for other important metrics of quality as well. 

The authors would like to acknowledge the internal funding from the Kaiser Permanente Southern California Care Improvement Research Team as well as the assistance of Ellen J. Rippberger, MPH, and Lorena Perez-Reynoso for their assistance with this work.

 Department of Research and Evaluation (ALS, ES, MKG), and Medical Director of Quality and Clinical Analysis (MHK), and Department of Regulatory Relations and Performance Assessment (LJB), Kaiser Permanente Southern California, Pasadena, CA; Department of Emergency Medicine, Los Angeles Medical Center, Kaiser Permanente Southern California, Los Angeles, CA.

 Internal funding from the KPSC Care Improvement Research Team (CIRT) supported this project. 

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

 Concept and design (ALS, ES, MHK); acquisition of data (LJB); analysis and interpretation of data (ALS, ES, MHK, MKG); drafting of the manuscript (ALS, ES, MHK); critical revision of the manuscript for important intellectual content (ALS, ES, MHK, LJB, MKG); statistical analysis (ES); administrative, technical, or logistic support (LJB); and supervision (ALS, MKG). 

 Adam L. Sharp, MD, MS, Kaiser Permanente Southern California, 100 S Los Robles Ave, Pasadena, CA 91101. E-mail: Adam.L.Sharp@kp.org. 

1. HCAHPS [Hospital Consumer Assessment of Healthcare Providers and Systems]: Patients’ Perspectives of Care Survey. CMS website. cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/HospitalHCAHPS.html. Published September 25, 2014. Accessed July 2, 2015.

2. Siegrist RB Jr. Patient satisfaction: history, myths, and misperceptions. 

. 2013;15(11):982-987. doi: 10.1001/virtualmentor.2013.15.11.mhst1-1311.

3. Kane RL, Maciejewski M, Finch M. The relationship of patient satisfaction with care and clinical outcomes. 

4. Jaipaul CK, Rosenthal GE. Do hospitals with lower mortality have higher patient satisfaction? a regional analysis of patients with medical diagnoses. 

. 2003;18(2):59-65. doi: 10.1177/106286060301800203.

5. Jayadevappa R, Schwartz JS, Chhatre S, Wein AJ, Malkowicz SB. Satisfaction with care: a measure of quality of care in prostate cancer patients. 

. 2010;30(2):234-245. doi: 10.1177/0272989X09342753.

6. Boulding W, Glickman SW, Manary MP, Schulman KA, Staelin R. Relationship between patient satisfaction with inpatient care and hospital readmission within 30 days. 

7. Zgierska A, Rabago D, Miller MM. Impact of patient satisfaction ratings on physicians and clinical care. 

. 2014;8:437-446. doi: 10.2147/PPA.S59077.

8. Fenton JJ, Jerant AF, Bertakis KD, Franks P. The cost of satisfaction: a national study of patient satisfaction, health care utilization, expenditures, and mortality. 

. 2012;172(5):405-411. doi: 10.1001/archinternmed.2011.1662.

Articles

CLINICAL

Antibiotics are not required to achieve high patient satisfaction. Patient age, chronic conditions, and an established patient–provider relationship influence satisfaction with acute care.

Low-Value Antibiotic Prescribing and Clinical Factors Influencing Patient Satisfaction

 In 2011, the California Right Care Initiative implemented a countywide physician organization learning collaborative called University of Best Practices (UBP) in San Diego County for major healthcare systems and physician organizations to share best practices in managing cardiovascular and cerebrovascular risk factors. Our objective was to examine whether UBP was associated with fewer hospitalizations for heart attacks and strokes.

 A quasi-experimental design was used to compare age-adjusted adult hospitalization rates before UBP initiation (2007-2010) against rates after UBP initiation (2011-2014) in San Diego County versus the rest of California.

 Difference-in-differences (DID) logistic regression models were estimated using hospitalization data from the California Office of Statewide Health Planning and Development for 2007 to 2014, including 372,205 and 642,455 hospitalizations for heart attacks and strokes, respectively.

 In the UBP versus pre-UBP period, the odds of adults being hospitalized for a heart attack in San Diego County decreased (odds ratio [OR], 0.84), whereas the odds stayed the same for adults in the rest of California (OR, 1.00): DID ratio of OR, 0.84 (

 <.001). This relative decrease was equivalent to 2735 (or 16.5%) fewer hospitalizations, totaling $61 million (2014 dollars). No robust association was found between UBP implementation and hospitalizations for strokes.

 A countywide physician organization learning collaborative was associated with fewer hospitalizations for heart attacks, but not for strokes. Healthcare systems and physician organizations should consider forming collaboratives to share best practices to manage patients’ cardiovascular and cerebrovascular risk factors, which may lead to fewer hospitalizations and reduced healthcare costs.

This study evaluated the California Right Care Initiative’s University of Best Practices (UBP) in

UBP is a countywide physician organization learning collaborative that shares best practices in managing cardiovascular and cerebrovascular risk factors. 

UBP was associated with lower age-adjusted hospitalization rates for heart attacks in San Diego County compared with the rest of California. 

Healthcare systems and physician organizations should consider forming collaboratives to share best practices to manage patients’ cardiovascular and cerebrovascular risk factors, which may lead to fewer hospitalizations and reduced healthcare costs.

Cardiovascular disease is the leading cause of death in the United States, and its annual costs, combined with those of stroke, averaged $316.6 billion per year in 2011 and 2012, including $193.1 billion in direct healthcare costs and $123.5 billion in indirect costs.

 Achieving lower heart attack and stroke rates will not only save lives, but also generate substantial healthcare cost savings.

Quality improvement collaboratives have become a well-established approach for improving the quality of care in the United States.

 Several national and regional models have emerged to reduce cardiovascular and cerebrovascular diseases and their risk factors. Million Hearts, for instance, an initiative launched by HHS in 2011, aims to prevent 1 million heart attacks and strokes over 5 years by convening and educating communities, physician groups, federal agencies, and other partners.

 Regional models that aim to reduce cardiovascular and cerebrovascular risk factors include physician collaboratives focused on education, training, and data sharing. For example, South Carolina’s Hypertension Initiative, which spread to 4 states in the stroke belt, contributed to a 43% decline in coronary heart disease deaths and a 42% decline in stroke deaths between 1996 and 2006.

 Another collaborative in Ontario, Canada, promoted education in 11 primary care clinical practices, which resulted in reductions in patients’ systolic blood pressure and 10-year cardiovascular risk scores during the 3-year study period.

 Finally, a community-wide cardiovascular disease prevention collaboration in rural Franklin County, Maine, was associated with lower hospitalization and mortality rates over a 40-year period.

The California Right Care Initiative (RCI) began in 2007, focusing on evidence-based and outcomes-improvement strategies to reduce preventable morbidity and mortality among Californians, particularly those with cardiovascular and/or cerebrovascular disease.

 One of RCI’s main strategies is to stimulate the sharing of best practices for managing risk factors (including diabetes) for these diseases among physician organizations, integrated delivery systems, and other stakeholders through regional inter-organizational learning communities known as University of Best Practices (UBP).

The San Diego County UBP (now called “Be There San Diego”) is the longest running RCI learning collaborative. San Diego County’s efforts began with a National Institutes of Health Grand Opportunity grant in October 2009, in which the California Department of Managed Health Care and physician leaders began to plan for the learning collaborative. The UBP meetings began in February 2011, and the monthly learning collaborative is attended by medical, pharmacy, and quality improvement directors from all of the major physician organizations, integrated delivery systems, and community clinics in the county. The attendees represent organizations that serve approximately 80% of San Diego County patients, including those who are commercially insured, Medicare, Medi-Cal, safety net, US Navy, and US Department of Veterans Affairs. Each meeting is devoted to presentations and discussions of better ways to care for patients with cardiovascular and cerebrovascular disease risk factors (eg, high blood pressure, high lipid levels, high blood sugar levels, diabetes, obesity, and smoking) through patient activation methods, healthcare team coordination, and uptake of medication protocols. Each organization reports and shares its progress on these measures because controlling these intermediate outcomes prevents unnecessary heart attacks and strokes.

The objective of this study is to examine whether the California RCI’s UBP in San Diego County was associated with fewer hospitalizations for acute myocardial infarctions (hereafter “heart attacks”) and cerebrovascular events (hereafter “strokes”). Learning collaboratives to improve coordination and quality of care are becoming more prevalent, as evidenced by recent initiatives including the CMS State Innovation Model Initiative, CMS’s Transforming Clinical Practice Initiative, the Medicaid program, and the Children’s Health Insurance Program.

 In spite of large-scale initiatives using regional learning collaboratives, no study has examined the impact of a physician organization learning collaborative in the context of a highly competitive managed care market that is attempting to collaborate on shared community-wide goals to better manage cardiovascular and cerebrovascular risk factors.

Inpatient hospitalization data are from California Office of Statewide Health Planning and Development’s (OSHPD) nonpublic Patient Discharge Data for 2007 to 2014 for adults 18 years or older. OSHPD collects these data from all hospitals in California. To identify relevant hospitalizations, we examined the principal discharge diagnosis code, which was based on the 

). We used code 410 to capture acute myocardial infarction (heart attack)

 and codes 430 to 438 to capture strokes.

 Hospitalizations were converted to age-adjusted rates, based on California’s 2014 population,

 using the direct standardization method for those aged 18 to 64 years and those 65 years or older.

During the 2007 to 2014 study period, there were 25,287,552 hospitalizations in California, including 437,774 for heart attacks and 667,776 for strokes in acute care hospitals. To avoid counting multiple hospitalizations for the same event, we excluded hospitalizations when a patient was discharged to the admitting or another hospital for acute inpatient care. These transfers likely occurred because the patient needed care at a higher-acuity hospital. This reduced the number of hospitalizations to 372,205 (—15.0%) for heart attacks and 642,455 (–3.8%) for strokes.

To estimate whether the California RCI’s UBP in San Diego County was associated with changes in hospitalization rates for heart attacks and strokes, we estimated difference-in-differences (DID) models, which are used in quasi-experimental research designs.

 A DID model reduces the potential for bias by controlling for baseline differences in hospitalization rates between San Diego County and the rest of California, and by controlling for hospitalization rate reductions that occurred in the whole state during this entire period. Equation 1: 

shows the logistic regression model used to estimate the DID models, where 

is 1 for a hospitalization for individual (

is 1 for a hospitalization in San Diego County and is 0 for a hospitalization in other counties; 

is 1 for years 2011 to 2014 and is 0 for years 2007 to 2010; 

are time-varying, county-level control variables from the California Health Interview Survey,

 including the percentage of the adult population that did not have health insurance, the percentage that did not have a usual place to go when sick or needing health advice, and the percentage that had an income from 0% to 199% of the federal poverty level because these factors are associated with the ability to pay for healthcare services and access to care. The DID interaction term compares how the odds of a hospitalization changed in the UBP period (2011-2014) versus the pre-UBP period (2007-2010) for San Diego County residents versus the residents in the rest of California. Separate models were estimated for heart attack and stroke hospitalizations.

In addition to the UBP in San Diego County, Sacramento and Los Angeles counties began UBPs in July 2012 and April 2013, respectively; however, they have far less physician organization participation and lower fidelity of implementation of learning collaborative models.

 As a sensitivity analysis, we re-estimated the DID models by excluding these counties.

We also analyzed pre-UBP hospitalization trends for San Diego County and the rest of California, because a DID model assumes parallel pretreatment trends, which means that absent the UBP intervention, the hospitalization rates in San Diego County and the rest of California are expected to change at the same rate.

 As a supplemental analysis, we extended the pre-UBP analysis by examining pre-UBP versus UBP period hospitalization trends in San Diego County and the rest of California using a piecewise linear regression difference-in-differences-in-differences (DIDID) model. The 

Our data only include individuals who were hospitalized. Therefore, we estimated the logistic regression models using Stata version 12 (StataCorp; College Station, Texas) 

command that implicitly includes individuals who were not hospitalized, because one variable within the 

command is the population, which we obtained from the California Department of Finance.

 The number of individuals who were not hospitalized equals the population minus the number of hospitalizations from individuals who were hospitalized, which allows for more than 1 hospitalization in a year for a separate medical event for an individual (eg, 

 code 412 is for previous heart attacks, which we did not include). The other key variable within the 

command is the age-adjusted number of hospitalizations, which we calculated by multiplying the aged-adjusted rate by the population for each county(s)-year. The 

command produces the same results as if we had estimated our models as logistic regressions with individual-level data that include both hospitalized and nonhospitalized individuals.

Our blogit model estimation approach is similar to 2 studies that analyzed emergency department (ED) visits, which explicitly included individuals who were not present in the ED visit data.

 We did not explicitly include individuals who were not present in our hospitalization data because we age-adjusted our hospitalization rates. Moreover, San Diego County did not experience gender and age demographic changes between the 2007-to-2010 period and the 2011-to-2014 period that substantively differed from changes in the rest of California.

To quantify the magnitude of our heart attack results from equation 1, we calculated how many additional hospitalizations would have occurred in San Diego County if there was no association between UBP and age-adjusted hospitalization rates. Furthermore, we calculated how many fewer hospitalizations would have needed to not occur in the rest of California if the same association between UBP and San Diego County’s age-adjusted hospitalization rates was also found in the rest of California. To translate hospitalizations into dollars, we assumed a hospitalization cost a payer $22,427 in 2014 inflation-adjusted dollars.

Our study was approved by the California Health and Human Services Agency’s and University of California, Berkeley’s institutional review boards.

 shows the age-adjusted number of hospitalizations per 100,000 adult population for heart attacks and strokes in San Diego County versus the rest of California between 2007 and 2014 (see 

 for the underlying data). During this period, the rate for heart attacks decreased sharply in San Diego County from 180.8 to 137.4 (24.0%), but the decrease was more moderate in the rest of California, from 185.5 to 160.8 (13.3%). For strokes, the rate decreased in San Diego County from 310.1 to 277.6 (10.5%), but the percentage decrease was larger in the rest of California, from 323.6 to 269.5 (16.7%). During the UBP period, the rate for heart attacks decreased more in San Diego County (23.5%) than in the rest of California (7.6%), but decreases for strokes were similar in San Diego County (13.8%) and the rest of California (12.3%).

 shows the age-adjusted number of hospitalizations per 100,000 adult population for heart attacks by gender, which shows the rate for males is approximately twice the rate of females, but both genders experienced similar decreases during the UBP period.

 shows the DID logistic regression results for heart attacks (models A1-A3) and strokes (models A4-A6). The top portion of the heart attack and stroke sections show the summary DID results based on the logistic regression models: they report each odds ratio (OR) that makes up the DID parameter, which is a ratio of 2 ORs. The summary result from equation 1 for San Diego County is 

. Below the summary DID results, the table reports the regression parameter estimates for each model. 

In the UBP versus pre-UBP period, the odds of adults being hospitalized for a heart attack in San Diego County decreased (OR, 0.84), while the odds stayed the same for adults in the rest of California (OR, 1.00): DID ratio of OR, 0.84 (

<.001) (model 1). The results were substantively the same for males and females (models 2 and 3, respectively). In the UBP period versus pre-UBP period, the odds of adults being hospitalized for a stroke in San Diego County decreased (OR, 0.95) slightly more than for adults in the rest of California (OR, 0.97), but the ratio of the ORs was not significant at the 0.05 level (OR, 0.98; 

= .08) (model 4). The stroke results were sensitive to inclusion of the control variables, whereas the heart attack results were not. For example, if the percentage of the adult population that reports no usual place of care is removed from the stroke model 4, then the DID ratio of OR increases to above 1.00 (DID ratio of OR, 1.05; 

<.001). It is important to control for this covariate, but the significant change from its exclusion makes the original result less robust.

Recall that a DID model assumes parallel pretreatment trends, which means that absent the UBP intervention, the hospitalization rates in San Diego County and the rest of California are expected to change at the same rate.

 Compared with the rest of California during the pre-UBP period, the San Diego County hospitalization trend approached being higher for heart attacks and was higher for strokes (see 

). Assuming the phenomenon driving the nonparallel trends continued into the UBP period and assuming UBP was associated with lower hospitalization rates, then San Diego County’s pre-UBP higher hospitalization trend would cause the DID results to be understated. This may be the case for heart attacks; however, for strokes, the pre-UBP trend differences are primarily because of San Diego County’s age-adjusted hospitalization rate in 2010 being an outlier. Its rate increased by 5.1% from 2009 to 2010, while the rest of California’s rate decreased by 0.3%. Therefore, it is difficult to know whether San Diego County’s 3.7% rate decrease in 2011 was because of UBP (ie, UBP reversed the pre-UBP trend difference) or if the 2011 decrease represented a regression to the mean. However, the DIDID results in 

 do not provide evidence to support that UBP was associated with a lower hospitalization rate trend for strokes during the UBP period: San Diego County’s UBP hospitalization trend (OR, 0.954; 

<.001) was similar to the rest of California’s (OR, 0.959; 

<.001), resulting in a UBP DID trend of 0.995 (

= .51) (see Model A10 in eAppendix Table 4).

Because of the difficulty of interpreting the DID parameter, a ratio of ORs, 

 presents the magnitude of the heart attack results from Table 1. From 2011 to 2014, there were 13,849 hospitalizations for heart attacks in San Diego County. Based on the logistic regression results in model 1, the difference between San Diego County’s and the rest of California’s age-adjusted hospitalization rate for heart attacks from 2007 to 2010 was 1.5 hospitalizations per 100,000 adults on average. If this difference had continued from 2011 to 2014, then there would have been 16,584 hospitalizations in San Diego County from 2011 to 2014. Therefore, this translates into the UBP being associated with 2735 hospitalizations being avoided in San Diego County (or 16.5% of the 16,584 potential) that would have cost payers $61 million (2014 dollars) in hospitalization costs from 2011 to 2014.

From 2011 to 2014, there were 174,195 hospitalizations for heart attacks in the rest of California (Table 2). Again, if the above rate difference of 1.5 had continued from 2011 to 2014, meaning that the rest of California experienced the UBP association with fewer hospitalizations for heart attacks, then there would have been only 153,994 hospitalizations in the rest of California from 2011 to 2014, a difference of 20,201 (11.6%) fewer hospitalizations at a cost of $453 million (2014 dollars).

Sacramento and Los Angeles counties began less-robust UBPs in July 2012 and April 2013, respectively. If these UBPs were associated with fewer hospitalizations, then they would have attenuated our original results because they were included in the control group. When the DID regression model in equation 1 was re-estimated excluding these counties, the heart attack result did not substantively change (model 1 re-estimate: DID ratio of OR, 0.85; 

<.001), but the stroke result actually became weaker (model 4 re-estimate: DID ratio of OR, 1.00; 

= .79), contributing to the original stroke result’s lack of robustness.

Furthermore, we conducted a falsification test and re-estimated our DID regression model in equation 1 for heart attacks (model 1 in Table 1) by treating the 8 counties with populations of more than 750,000 adults in 2014 as though they received the UBP treatment beginning in 2011. In summary, we did not find consistent evidence that any of these 8 counties experienced what San Diego County did. Although Alameda and Orange counties’ DID results were lower or similar to San Diego County’s, those results were partially caused by their pre-2011 lower hospitalization rate trend compared with the rest of California (see eAppendix for details).

The California RCI’s UBP in San Diego County, a countywide physician organization learning collaborative, was associated with a lower level of age-adjusted adult hospitalization rates for heart attacks compared with the rest of California. The UBP was associated with 2735 (16.5%) fewer hospitalizations for heart attacks, totaling $61 million (2014 dollars). If the rest of California had experienced the same reduction in hospitalizations for heart attacks associated with UBP, then there would have been 20,201 (11.6%) fewer hospitalizations at a cost of $453 million (2014 dollars). This result is consistent with other regional collaborative efforts to improve hypertension care

 and inter-organizational learning activities being associated with performance improvement.

 However, a similar robust association between UBP and hospitalization rates for strokes was not found.

Our findings extend the current literature by providing evidence that physician organizations operating in the context of a highly competitive managed care market can collaborate and exchange best practices to achieve shared community-wide goals to decrease hospitalizations for heart attacks. Although there is limited evidence about the relative impact of various learning collaborative components, recent research has identified the core components of learning collaboratives that participants value most.

 Collaborative faculty, solicitation of staff ideas, change package, Plan-Do-Study-Act cycles, facilitated learning sessions, and a collaborative extranet have been identified as collaborative components that provide participants with motivation, social support, and project management skills.

 These components leverage intrinsic motivation among physicians for quality improvement.

 The UBP collaborative incorporated these core components directly or with modification. Our findings suggest that a combination of components may be needed in collaboratives to achieve desired performance outcomes as opposed to any single component. Future research should attempt to identify the key combinations of components that have the greatest impact for a particular context. For example, the UBP’s emphasis on the use of collaborative faculty (including a cardiology expert) and facilitated learning sessions warrant further study to determine if these 2 components are among the most important for achieving successful outcomes.

Although a quasi-experimental research design using DID models is a strong design, results could be biased if another intervention or phenomenon occurred contemporaneously with the UBP that was also associated with hospitalization rates in San Diego County and/or the rest of California. Kaiser Permanente has started a number of statewide initiatives in California to improve cardiovascular and cerebrovascular care, such as the Prevent Heart Attacks and Strokes Everyday (“PHASE”) program and the Aspirin, Lisinopril and Lipid-Lowering Medication (“ALL”) initiative; however, we do not think these would bias our results because they began well before UBP’s start in 2011.

 Changes in the share of patients who experienced a heart attack, but did not survive to be admitted into the hospital, could also bias our results, but there is no reason to think this change would have occurred disproportionately in San Diego County. Also, stroke hospitalizations were more difficult to analyze because San Diego County’s 2010 age-adjusted hospitalization rate for strokes was a high outlier: its rate increased by 5.1% that year, just before the start of UBP, whereas the rest of California’s rate decreased by 0.3%. This could be one reason why the stroke findings were not significant and sensitive to different model specifications. Finally, we do not think hospital closures significantly contributed to lower hospitalization rates in San Diego County, because among the 16 hospitals in the county in 2007, only Fallbrook Hospital, a small, 47-bed hospital, closed during the study period, but not until November 2014. On the other hand, Palomar Medical Center, a 288-bed hospital, opened in August 2012.

Our study results suggest California RCI’s UBP physician organization learning collaborative in San Diego County was significantly associated with—and likely at least a partial cause of&mdash;a marked decline in the number of hospitalizations for heart attacks. No robust relationship was found for hospitalizations for strokes. Although our findings are not explicitly generalizable to other physician organization learning collaboratives, they could inform similar existing or new learning collaboratives, whose numbers are increasing.

 As the Medicare Access and CHIP Reauthorization Act expands value-based payment arrangements, physicians and their affiliated hospitals will have greater incentives to test different learning collaborative models to reduce hospitalizations via prevention and disease management strategies. Thus, efforts to extend learning collaborative models to other counties in California and elsewhere in the United States become especially important.

 School of Public Health (BDF, SLI, HPR, SMS), and Haas School of Business (SMS), University of California, Berkeley, CA.

 This study was funded by the California Right Care Initiative (RCI). 

MANAGERIAL

The University of Best Practices physician organization learning collaborative in San Diego County was associated with lower hospitalization rates for heart attacks.

Countywide Physician Organization Learning Collaborative and Changes in Hospitalization Rates

Using a large natural experiment among 39 employers, we examined the effect of adding financial incentives to workplace wellness programs. 

 The 39 study employers used the same national insurer to administer their wellness programs, allowing us to observe preventive and health-promoting behaviors before and after financial incentives were implemented. Fifteen treatment employers introduced financial incentives into their wellness programs over 3 years, providing variation in the start dates, whereas 24 employers did not introduce financial incentives. These incentives were attached to specific health actions, including annual preventive visits, biometric screening, and selected screening services for diabetes, heart disease, and cancer. 

 Using multivariate regression, we examined employees and their adult dependents who had insurance coverage for at least 12 months and were offered a wellness program. Outcomes include utilization of annual preventive visits, low-density lipoprotein cholesterol testing, fasting blood sugar (FBS) testing, and breast, cervical, and colon cancer screens.

 Financial incentives increased annual preventive visits by 7.7 percentage points, cholesterol testing by 7.9 percentage points, and FBS testing by 7.1 percentage points (

<.05 for each). Compared with baseline rates, these changes represent significant improvements of 21% to 29%. Increases for cancer screening were smaller: 2.7 percentage points for mammograms and 2.2 percentage points for colorectal cancer screening, which correspond to increases over baseline rates of 5.5% and 7.3%, respectively. We did not detect an impact on cervical cancer screening. 

The addition of financial incentives to wellness programs increases their impact on selected preventive care services. 

The addition of financial incentives to workplace wellness programs has a notable impact on whether individuals receive preventive care services. Modest financial incentive programs in workplace settings can be effective; however, individuals who did not receive services in the past year respond less than others. Because targeting financial incentives to selected subgroups is challenging within the Affordable Care Act framework, wellness programs may require additional outreach efforts.Several strategies have been suggested to promote use of preventive services, including expanded incentives and greater involvement of employers.

 Under the Affordable Care Act (ACA), new incentives were created to promote employer wellness programs and encourage employers and employees to support healthier workplaces. The law enables employers to offer financial rewards to employees who participate in wellness programs or meet certain health-related targets.

 Increasingly, employers offering wellness programs are incorporating these financial incentives with the belief that they will boost the impact of their programs; however, little is known about the effectiveness of these incentive programs. 

In 2015, 14% of all employers, half of employers with 200 or more employees, and two-thirds of employers with 1000 or more employees offered wellness programs.

 These programs include health screenings that collect self-reported health risk information and biometric data from an in-person health examination conducted by a medical professional.

 Many employers incorporate financial incentives into their wellness programs. Thirty-two percent of employers with biometric screening programs incorporate a financial incentive for employees who complete the biometric screens; among large employers (≥200 employees) with wellness programs, 56% do so.

The use of incentives in conjunction with wellness programs is largely driven by employers’ beliefs that programs with incentives are somewhat (64% of employers who use incentives) or very effective (27%). Incentives are disbursed in a variety of ways, including premium discounts, waivers of cost sharing, or additional covered benefits. In practice, typical incentives range from $150 to $500, while the average annual premium for single coverage is $6251.

Despite their widespread use, systematic evidence about the effectiveness of wellness programs with financial incentives is lacking. One literature review concluded that incentives were effective at increasing participation in self-reported health assessments, but was unable to assess their impact on the completion of preventive screenings.

 In randomized controlled settings, financial rewards not paired with wellness programs have been effective at increasing participation in health risk assessment completion,

 Randomized trials provide significant proof of concept that targeted financial incentives can be effective; however, as with all randomized trials in which participation requires an opt-in consent, it is difficult to assess generalizability, as many such trials only enroll 10% to 15% of potentially eligible individuals and participants likely differ from those who do not volunteer to participate on unmeasured characteristics, such as motivation.

The current study takes advantage of a sizeable natural experiment in which 39 large employers within the United States initially offered wellness programs without financial incentives. Over time, 15 of them added financial incentives and the remaining 24 did not. Financial incentives were attached to specific health actions, including annual preventive visits, biometric screening, and selected screening services for diabetes, heart disease, and cancer. Employees received personalized scorecards, both online and mailed to their home, to help them track their progress toward receiving incentives, which were awarded as premium reductions, cash, or gift cards. To our knowledge, ours is the largest study of wellness programs with financial incentives to date and includes over 1.4 million insured enrollees. A key strength of our design is the ability to examine pre-post effects of the introduction of incentives with a contemporaneous wellness program control group that did not introduce incentives. 

In addition, we considered whether the financial incentive effect is greatest for nonregular users of preventive services or for users who had received the service in the prior year. The financial incentive program is designed to be broad-based rather than targeted. Nonetheless, a program is more economically efficient if it entices new or nonregular users of prevention services rather than rewarding individuals who likely would have received the services in any case. 

The wellness program we studied allows enrollees to earn dollars or points for adopting better health behaviors. Enrollees are provided a personalized scorecard, which includes health actions that were completed, as well as those that were not, as an aid to maintain or improve their health behaviors. The points earned for various activities can be converted into cash rewards, premium reductions, or gift cards in a manner set by the employer. We examined clinical screening outcomes that can be measured via claims data. The incentive for any given clinical screening was the same for all covered individuals within an employer, but could vary across employers. Incentive amounts ranged from $0 to $80 for preventive visits, $0 to $100 for low-density lipoprotein cholesterol (LDL-C) tests, $0 to $100 for blood sugar ascertainment with glycated hemoglobin, and $0 to $75 for cancer screening tests. The maximum annual award that an individual could earn for receiving all of these services ranged from $250 to $900 across employers. By contrast, employers without financial incentives in their wellness program promoted and measured the same outcomes, but employees did not receive an explicit financial reward.

All employers used the same insurer to administer their wellness programs, allowing us to observe preventive and health promoting behaviors before and after financial incentives were implemented. The 15 treatment employers introduced financial incentives into their wellness programs between 2010 and 2012, providing variation in the start dates. Individuals’ outcomes were observed as long as they were covered by the insurer. In many cases, the insurer was the sole provider of coverage for the employers. The staggered implementation by employer is illustrated in the 

. Our data span January 2009 through December 2013. 

We used de-identified healthcare claims and health plan enrollment and wellness program data from the insurer. Outcome variables were obtained from claims data based on standard claims codes. These include utilization of annual preventive visits, LDL-C tests, fasting blood sugar tests, and breast, cervical, and colon cancer screens. Coding details are provided in 

]. We were able to determine chronic conditions from claims, but not body mass index (BMI). From enrollment data, we obtained individual age, gender, dates and type of coverage, and whether the insured member was an employee or adult dependent. Insurance enrollment data typically have limited demographic information. However, through a vendor, the insurer obtained imputed information on race, ethnicity, and education of covered members, all potentially important factors that could influence an individual’s propensity to seek preventive care. 

Our sample was restricted to adult (aged 18-64 years) employees who were covered by the insurer administering the wellness programs for at least 12 continuous months (1 full plan year). Spouses and domestic partners were included if they, in addition to the employee, were offered the wellness program. All sample members were included in models that examined annual preventive visits, LDL-C screening, and fasting blood sugar screening. Different subgroups were examined for each preventive screening (ie, women aged 40-64 years for breast cancer, women aged 18-64 years for cervical cancer, and men and women aged 50-64 years for colorectal cancer). Individuals older than 64 years were excluded because they were eligible for Medicare coverage. 

Our study employed a panel-data difference-in-differences (DID) design in which 39 employers offered wellness programs without financial incentives at baseline, 15 employers added financial incentives and the remaining 24 comparison employers did not.

 This is also commonly referred to as a stepped-wedge design. Because all employers used the same national insurer to administer their wellness programs, we observed preventive and health promoting behaviors before and after financial incentives were implemented based on common data collection. 

Financial incentives created under the Affordable Care Act can help promote employer wellness programs and support preventive services utilization.

October 2017

Low-Value Antibiotic Prescribing and Clinical Factors Influencing Patient Satisfaction

Countywide Physician Organization Learning Collaborative and Changes in Hospitalization Rates

Boosting Workplace Wellness Programs With Financial Incentives

Use of Patient-Reported Outcomes and Satisfaction for Quality Assessments

Trends in Hospital–Physician Integration in Medical Oncology

Improving Care Transitions: Complex High-Utilizing Patient Experiences Guide Reform

The Option Value of Innovative Treatments for Non–Small Cell Lung Cancer and Renal Cell Carcinoma

Statewide Data Infrastructure Supports Population Health Management: Diabetes Case Study

Is Higher Patient Satisfaction Associated With Better Stroke Outcomes?

A Health Plan's Investigation of Healthy Days and Chronic Conditions