Currently Viewing:
The American Journal of Managed Care November 2017
Using the 4 Pillars to Increase Vaccination Among High-Risk Adults: Who Benefits?
Mary Patricia Nowalk, PhD, RD; Krissy K. Moehling, MPH; Song Zhang, MS; Jonathan M. Raviotta, MPH; Richard K. Zimmerman, MD, MPH; and Chyongchiou J. Lin, PhD
The Influence of Provider Characteristics and Market Forces on Response to Financial Incentives
Brock O’Neil, MD; Mark Tyson, MD; Amy J. Graves, SM, MPH; Daniel A. Barocas, MD, MPH; Sam S. Chang, MD, MBA; David F. Penson, MD, MPH; and Matthew J. Resnick, MD, MPH
Patients' Perspectives of Care Management: A Qualitative Study
Ann S. O’Malley, MD, MPH; Deborah Peikes, PhD, MPA; Claire Wilson, PhD; Rachel Gaddes, MPH; Victoria Peebles, MSW; Timothy J. Day, MSPH; and Janel Jin, MSPH
Impact of Health Reform on Young Adult Prescription Medication Utilization
Amy Pakyz, PharmD, PhD, MS; Hui Wang, PhD; and Peter Cunningham, PhD
Reframing the Unaffordability Debate: Patient Responsibility for Physician Care
Katherine Hempstead, PhD; Josh Gray, MBA; and Anna Zink, BA
Electronic Reminder's Role in Promoting Human Papillomavirus Vaccine Use
Jaeyong Bae, PhD; Eric W. Ford, PhD, MPH; Shannon Wu, BA; and Timothy Huerta, PhD, MS
Currently Reading
Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial
Adam L. Sharp, MD, MS; Yi R. Hu, MS; Ernest Shen, PhD; Richard Chen, MD; Ryan P. Radecki, MD, MS; Michael H. Kanter, MD; and Michael K. Gould, MD, MS
Diabetes Care Improvement in Pharmacist- Versus Nurse-Supported Patient-Centered Medical Homes
Lillian Min, MD, MSHS; Christine T. Cigolle, MD, MS; Steven J. Bernstein, MD, MPH; Kathleen Ward, MPA; Tisha L. Moore, MPH; Jinkyung Ha, PhD; and Caroline S. Blaum, MD, MS
Validation of a Claims-Based Algorithm to Characterize Episodes of Care
Chad Ellimoottil, MD, MS; John D. Syrjamaki, MPH; Benedict Voit, MBA; Vinay Guduguntla, BS; David C. Miller, MD, MPH; and James M. Dupree, MD, MPH

Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial

Adam L. Sharp, MD, MS; Yi R. Hu, MS; Ernest Shen, PhD; Richard Chen, MD; Ryan P. Radecki, MD, MS; Michael H. Kanter, MD; and Michael K. Gould, MD, MS
The authors assessed the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis. Education and CDS improved antibiotic stewardship and changed diagnosis patterns.

Objectives: Antibiotic stewardship is key to optimizing patient outcomes and affordable care. The study objective was to examine the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis among providers of varying experience.

Study Design: A stepped-wedge cluster randomized intervention to evaluate antibiotic use for acute sinusitis encounters at 126 Kaiser Permanente Southern California clinics between September 2014 and April 2015.

Methods: The primary outcome was receipt of an antibiotic prescription. Multivariate analysis adjusted for patient, provider, and medical center characteristics. Secondary analyses described sinusitis and other common upper respiratory infection (URI) diagnoses and antibiotic use during the study period compared with prior years. Chart review of a random sample reported the proportion of encounters receiving guideline-concordant antibiotics. 

Results: Analysis of 21,949 encounters (10,491 pre- and 11,458 post intervention) showed CDS reduced the use of antibiotics (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87), although the pre-post absolute difference was small (85.9% vs 83.9%, respectively). Education had a large initial effect (AOR, 0.51; 95% CI, 0.46-0.57), which did not persist. Increasing years of provider experience raised the rates of antibiotic prescribing, but did not have a significant interaction with CDS (P = .19). The effect of CDS varied by medical center (P <.001). In addition, sinusitis diagnoses decreased post intervention, with no overall increase in antibiotic prescribing for URI diagnoses. Lastly, guideline-concordant antibiotic use increased by 14%. 

Conclusions: Provider education and CDS improved antibiotic stewardship and changed diagnosis patterns. The benefits of education were brief, and CDS effectiveness varied by medical center. 

Am J Manag Care. 2017;23(11):e360-e365
Takeaway Points

It is vital to understand the effectiveness of interventions designed to improve patient care in order to disseminate and replicate successful initiatives and avoid fruitless care improvement efforts. Our study of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis offers the following to inform health systems and future research: 
  • CDS integrated in an electronic health record can discourage inappropriate antibiotic prescribing and will likely change diagnostic patterns. 
  • Provider education is unlikely to have a sustained effect on antibiotic prescribing.
  • The effectiveness of education and CDS will vary based on inherent differences among locations.
Antimicrobial stewardship is key to optimizing patient outcomes and minimizing the emergence of drug-resistant bacterial organisms.1-3 Unfortunately, the goal of reducing inappropriate antibiotic prescribing has remained elusive.4 This is true despite the existence of a variety of strategies to improve antibiotic stewardship, such as pay-for-performance incentives, documentation change strategies, provider education, and social “nudges.”5-7 We currently do not understand why these approaches have variable results across different settings. Research is needed to describe the effect of such interventions in general community practice and to understand variations in effect due to patient, provider, or medical center characteristics in order to inform current efforts to improve care. 

Acute sinusitis is a condition that affects 31 million annually8 and costs billions.9 The preponderance of evidence suggests antibiotics are not only ubiquitously overused for acute sinusitis treatment,10-13 but the correct antibiotic is often not chosen.14 Despite agreement regarding overuse, there is no universal recommendation regarding the precise group of patients with acute sinusitis who may benefit from antibiotics.8,15-17 We used the recommendations from our health system, patterned after the Infectious Diseases Society of America (IDSA)’s guideline,16 and a practical study design18 to report the effects of a 2-component intervention designed to improve antibiotic stewardship for acute sinusitis encounters. The components were computerized clinical decision support (CDS) within an electronic health record (EHR)19,20 and provider education.1 

Clinical leaders and policy makers need to understand the variable effectiveness of multifaceted interventions, as well as the importance that individual provider characteristics may have on expected results. Our aim was to enhance the understanding of the effects of clinician education and CDS in a real-life clinical setting and, specifically, to assess any differences in effectiveness on providers with varying years of experience or among medical centers. 


We performed a pragmatic stepped-wedge cluster randomized study of provider education and CDS for acute sinusitis encounters. The study was performed in outpatient clinics (105 primary care, 21 urgent care) between September 2014 and April 2015 within Kaiser Permanente Southern California (KPSC), a large integrated health system. This system provides healthcare to more than 4 million members representative of the population diversity found in southern California. We used information from the EHR collected during routine care. This study was approved by the KPSC Institutional Review Board.

Each participating primary care and urgent care clinic is clustered within 1 of 6 medical service areas. Each service area was randomly assigned to receive the CDS intervention as part of a staged implementation over the course of 6 months. The total study period was 8 months, allowing for 1 month of pre-data for all areas prior to CDS implementation. CDS was turned on in 1 of the 6 areas each month until all were receiving CDS, then 1 month of post data was collected after all had received the intervention (Figure 1). This staggered implementation allowed for a stepped-wedge analysis.21 This design controls for confounders due to secular trends and strong champion effects. 

CDS was based on qualitative interviews with practicing clinicians22 and designed using features shown to be effective in previous trials.19 Automated in the EHR, CDS was initiated at the time of antibiotic prescribing for patients with an acute sinusitis diagnosis. Once triggered, it offered a clear recommendation based on current evidence.16 The alert informed providers that antibiotics were not generally recommended for patients with symptoms of less than 10 days’ duration. Additionally, CDS included options to justify use of antibiotics in the case of prolonged symptoms, “double worsening” of symptoms, or abnormal physical exam findings. CDS could be bypassed by providers, but required a response or cancellation to proceed with the order. In addition, CDS directed providers to an order set to facilitate the prescribing of amoxicillin or amoxicillin clavulanate if they had ordered a nonrecommended antibiotic for acute sinusitis. 

Provider education included a recorded online presentation and 2 webinars delivered in the middle of the staggered rollout (December). The intervention was created after previous research showed the need for improved stewardship and no temporal trends in improvement in the years leading up to our study period.13 

All acute sinusitis encounters (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] code 461.x) taking place in primary care settings were included. We excluded encounters with specialists, follow-up visits, and patients with chronic diseases compromising their immune response (chronic liver disease, ICD-9-CM code 571; end-stage renal disease, ICD-9-CM code 585.6; congestive heart failure, ICD-9-CM code 428; immune disorders, ICD-9-CM code 279; malignant neoplasms, ICD-9-CM codes 140-165, 170-176, 179-209, and 235-239; and common rheumatologic disorders, ICD-9-CM codes 714, 710, 555.9, and 556). To ensure completeness of data, we limited our sample to Kaiser health plan members. 

The primary outcome was prescription of an antibiotic, classified by the generic product identifier. Secondary outcomes included prescription of an antibiotic for diagnoses with similar symptoms as acute sinusitis (acute nasopharyngitis, ICD-9-CM code 460; nonstreptococcal pharyngitis, ICD-9-CM code 462; acute laryngitis, ICD-9-CM code 464; acute laryngopharyngitis, ICD-9 code 465; acute upper respiratory tract infections of other multiple sites, ICD-9-CM code 465.8; acute upper respiratory tract infections not otherwise specified, ICD-9-CM code 465.9; acute bronchitis, ICD-9 code 466; influenza with respiratory manifestations, ICD-9-CM code 487.1; and bronchitis not specified as acute or chronic, ICD-9-CM code 490)7 and concordance with current IDSA16 guidelines.

Concordance estimates were done based on structured chart review from a random sample of 50 charts with some from each of the 6 medical service areas. The structured chart review was performed by a trained member of the research team who followed a standardized protocol to limit bias.23,24 This random sample was selected from encounters in which antibiotics were prescribed post intervention. We used a standardized data abstraction form and manual previously developed to identify guideline-concordant antibiotic use for similar acute sinusitis encounters, which is proven to have high interrater reliability (93% agreement, kappa = .86).13 This previous research provided estimates of concordance rates (32%) for acute sinusitis encounters receiving antibiotics for comparison purposes. We used results from our chart review and applied the sampling fraction to estimate the variance of the proportion of encounters receiving antibiotics concordant with recommendations post intervention. This estimate was calculated via standard formulas routinely used for survey sampling research,25 then applied to the entire sample to construct 95% confidence limits. 

We used multiple logistic regression analyses to examine the main effect of CDS on antibiotic prescribing. To account for the staggered implementation of CDS, a dichotomous variable was created to indicate whether the encounter was pre- or post CDS. Another indicator variable was included to assess the impact of education amidst the rollout of CDS. A facility variable was included to identify differences in effect among the service areas. The analysis adjusted for variables that were statistically associated (P <.05) with receipt of antibiotics. 

The final model used the provider as a random intercept, the education indicator as a fixed effect, and indicator variables modeled as fixed effects to assess the separate effects of CDS and provider education. 

Secondary analyses examined the interaction between CDS and provider groups of varying years of experience. We categorized the years of provider experience into 4 groups (<3 years, 3-9 years, 10-20 years, and >20 years), using the largest group of providers (10-20 years) as the reference for comparison. We similarly examined the interaction between CDS and each of the medical service areas. Within these groups of providers and service areas we also compared the odds of receiving antibiotics pre- and post intervention. 

We performed additional analyses to understand changes in sinusitis encounter diagnoses compared with similar upper respiratory infections (URIs) and to evaluate if these encounters resulted in changes to antibiotic prescribing. We report the number of acute sinusitis encounters and other similar URIs, as well as the proportion resulting in antibiotics during the study period (September 2014-April 2015) compared with the same months in the 2 prior years, 2012 to 2013 and 2013 to 2014. Similarly, we report these diagnoses and the percent receiving antibiotics stratified pre-post CDA implementation and pre-post provider education during the study period, as well as the type of antibiotic used pre-post CDS for acute sinusitis encounters. 


The study sample included 21,949 initial acute sinusitis encounters for adults (Figure 1). A similar number of encounters were included before and after the implementation of CDS (10,491 pre- and 11,458 post intervention), and more patients were seen at primary care clinics (76%) than in urgent care settings (Table 1). 

Primary Outcome

The CDS intervention significantly reduced antibiotic use for acute sinusitis encounters (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87) (Table 2), although it demonstrated only a 2% absolute reduction. Provider education, assessed as a time effect during December, showed significant decreases in antibiotic prescribing (aOR, 0.51; 95% CI, 0.46-0.57) (Table 2), but these effects were not sustained over the study period (Figure 2). 

Secondary Outcomes

The odds of receiving antibiotics increased with additional years of provider experience (Table 2), although there was not a statistically significant interaction between years of provider experience and CDS (P = .19) (Table 3). There were significant differences in the effect of CDS and antibiotic prescribing based on the interaction with each medical service area (P <.001).

Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up