Currently Viewing:
The American Journal of Managed Care November 2017
Using the 4 Pillars to Increase Vaccination Among High-Risk Adults: Who Benefits?
Mary Patricia Nowalk, PhD, RD; Krissy K. Moehling, MPH; Song Zhang, MS; Jonathan M. Raviotta, MPH; Richard K. Zimmerman, MD, MPH; and Chyongchiou J. Lin, PhD
The Influence of Provider Characteristics and Market Forces on Response to Financial Incentives
Brock O’Neil, MD; Mark Tyson, MD; Amy J. Graves, SM, MPH; Daniel A. Barocas, MD, MPH; Sam S. Chang, MD, MBA; David F. Penson, MD, MPH; and Matthew J. Resnick, MD, MPH
Patients' Perspectives of Care Management: A Qualitative Study
Ann S. O’Malley, MD, MPH; Deborah Peikes, PhD, MPA; Claire Wilson, PhD; Rachel Gaddes, MPH; Victoria Peebles, MSW; Timothy J. Day, MSPH; and Janel Jin, MSPH
Impact of Health Reform on Young Adult Prescription Medication Utilization
Amy Pakyz, PharmD, PhD, MS; Hui Wang, PhD; and Peter Cunningham, PhD
Reframing the Unaffordability Debate: Patient Responsibility for Physician Care
Katherine Hempstead, PhD; Josh Gray, MBA; and Anna Zink, BA
Electronic Reminder's Role in Promoting Human Papillomavirus Vaccine Use
Jaeyong Bae, PhD; Eric W. Ford, PhD, MPH; Shannon Wu, BA; and Timothy Huerta, PhD, MS
Currently Reading
Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial
Adam L. Sharp, MD, MS; Yi R. Hu, MS; Ernest Shen, PhD; Richard Chen, MD; Ryan P. Radecki, MD, MS; Michael H. Kanter, MD; and Michael K. Gould, MD, MS
Diabetes Care Improvement in Pharmacist- Versus Nurse-Supported Patient-Centered Medical Homes
Lillian Min, MD, MSHS; Christine T. Cigolle, MD, MS; Steven J. Bernstein, MD, MPH; Kathleen Ward, MPA; Tisha L. Moore, MPH; Jinkyung Ha, PhD; and Caroline S. Blaum, MD, MS
Validation of a Claims-Based Algorithm to Characterize Episodes of Care
Chad Ellimoottil, MD, MS; John D. Syrjamaki, MPH; Benedict Voit, MBA; Vinay Guduguntla, BS; David C. Miller, MD, MPH; and James M. Dupree, MD, MPH

Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial

Adam L. Sharp, MD, MS; Yi R. Hu, MS; Ernest Shen, PhD; Richard Chen, MD; Ryan P. Radecki, MD, MS; Michael H. Kanter, MD; and Michael K. Gould, MD, MS
The authors assessed the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis. Education and CDS improved antibiotic stewardship and changed diagnosis patterns.
The overall number of encounters with acute sinusitis diagnoses decreased compared with the same temporal period in the 2 prior years and represented a smaller proportion of acute URI diagnoses (8.1% vs 11.7% and 11.3%). The proportion of acute sinusitis encounters, compared with all similar URI diagnoses, decreased in the posteducation (7.5% vs 9.3%) and the post-CDS (7.1% vs 9.2%) periods compared with pre-intervention. The overall proportion of acute URI encounters receiving antibiotics decreased during the study period (27.1%) compared with prior years (31.2% and 30.7%), but this does not appear to be attributable to the intervention. Pre-post stratification showed there may have been some diagnosis shifting where antibiotics were still prescribed, as CDS (3.3% increase) and education (2% increase) showed a small increase in antibiotic prescribing for other URIs before and after implementation during the study period (eAppendices 1 and 2 [eAppendices available at]). The types of antibiotics used pre-post intervention did not differ significantly (eAppendix 3).

Lastly, a structured review of acute sinusitis encounters for patients who received antibiotics found that 46% (95% CI, 32%-60%) were guideline concordant. This was a 14% absolute improvement from a structured review of acute sinusitis encounters from the same primary care clinics in the pre-implementation period (32%; 95% CI, 19%-45%).13 


We performed an effectiveness study of the ability of CDS and provider education to decrease the use of antibiotics for acute sinusitis encounters. Adjusted comparisons showed a 22% improvement in the odds of prescribing antibiotics after the intervention, but the absolute reduction was small (2%). After CDS and education, the number of encounters with an acute sinusitis diagnosis decreased substantially. Lastly, we found that CDS had variable effectiveness based on the medical service area.

This study makes several important contributions to the existing body of literature describing the effectiveness of CDS. First, we confirmed that CDS is effective in real-world clinical settings, but showed that the magnitude of benefit may be less than that observed in highly controlled studies of efficacy.26 

Second, we observed a pattern of diagnosis-shifting as a result of education and CDS implementation. This may be an example of documented CDS “workarounds,” and future studies and quality-improvement efforts should account for changes in diagnosis patterns.27 It is possible that providers substituted a diagnosis other than acute sinusitis once they were made aware that antibiotics were not indicated for most patients with sinusitis. Understanding how CDS modifies diagnosis patterns warrants future investigation, although, based on our results, these changes did not result in an overall increase in the use of antibiotics for all URI diagnoses. 

Third, we observed increasing odds of antibiotic prescribing correlated with increasing years of provider experience, but this did not result in varying CDS impact. CDS did vary in effectiveness, however, based on the medical service area of the encounter. This may be a result of the culture and context within particular settings that promotes success. This finding supports other reports that have shown these less tangible factors to be an important aspect of the success of interventions.28,29 What is clear is that clinical leaders, policy makers, and researchers should account for the variable effectiveness of CDS depending on the local medical center characteristics. 

Despite overall results demonstrating improvements in antibiotic stewardship and guideline-concordant prescribing, we observed a diminution in effect over time. Much of the overall effectiveness of the 2-component intervention may be attributed to the acute drop in prescribing in December associated with provider education, rather than the CDS intervention. This temporal improvement was not sustained, and by the end of the study period rebounded to near the prior baseline. This reinforces that sustained improvement in clinical practice is difficult to achieve. 

Our study confirms results from recent efficacy trials showing the benefit of “accountable justification”30 with automated CDS during outpatient encounters, although in a slightly different approach. The limited effectiveness of our results may be partially due to limited “accountability,” as our CDS required a clicked response, but did not require the provider to document in the medical record the reason for use of antibiotics. Additionally, we looked only at acute sinusitis encounters, a condition for which antibiotics are generally not indicated, except under special circumstances, instead of viral diagnoses for which antibiotics are always contraindicated. Based on our relatively large aOR and comparatively small absolute effect of CDS, we hypothesize that patients with shorter symptoms or less severe symptoms were those changed to a different diagnosis and not given antibiotics. This is also based on our chart review, which showed improvements in guideline-concordant use of antibiotics. This may explain the increase in other similar URI diagnoses without a rise in antibiotics for those conditions. Therefore, it is reasonable to predict that the effect of our intervention on overall use of antibiotics may be underestimated by the absolute 2% decrease in the proportion of encounters receiving antibiotics for acute sinusitis. 


Provider education and integrated CDS reduced antibiotic use for acute sinusitis encounters and substantially changed patterns of diagnosis for acute URIs. The benefits of education were brief and did not persist through the study period, and CDS effectiveness varied by medical center.


The authors thank Ellen J. Rippberger and Lorena Perez-Reynoso for their administrative assistance and organization throughout this study. They also recognize Jason Doctor, PhD, and Daniella Meeker, PhD, for reviewing and providing feedback about the manuscript. 

Author Affiliations: Department of Research and Evaluation, Kaiser Permanente Southern California (ALS, YRH, ES, MKG), Pasadena, CA; Department of Emergency Medicine, Los Angeles Medical Center, Kaiser Permanente Southern California (ALS), Los Angeles, CA; Department of Emergency Medicine, San Diego Medical Center, Kaiser Permanente Southern California (RC), San Diego, CA; Department of Emergency Medicine, Kaiser Permanente Northwest (RPR), Portland, OR; Medical Director of Quality and Clinical Analysis, Kaiser Permanente Southern California (MHK), Pasadena, CA.

Source of Funding: Internal funding from the KPSC Care Improvement Research Team (CIRT) supported this project.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

Authorship Information: Concept and design (ALS, ES, RC, MHK); acquisition of data (YRH); analysis and interpretation of data (ALS, YRH, ES, RC, RPR, MHK, MKG); drafting of the manuscript (ALS, ES, RPR); critical revision of the manuscript for important intellectual content (ALS, ES, RPR, MHK, MKG); statistical analysis (YRH, ES); administrative, technical, or logistic support (RC); and supervision (ALS).

Address Correspondence to: Adam L. Sharp, MD, MS, Kaiser Permanente Southern California, 100 S Los Robles Ave, 2nd Fl, Pasadena, CA 91101. E-mail: 

1. Society for Healthcare Epidemiology of America; Infectious Diseases Society of America; Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol. 2012;33(4):322-327. doi: 10.1086/665010.

2. Fishman N. Antimicrobial stewardship. Am J Med. 2006;119(6 suppl 1):S53-S61; discussion S62-S70. doi: 10.1016/j.amjmed.2006.04.003.

3. Roberts RR, Hota B, Ahmad I, et al. Hospital and societal costs of antimicrobial-resistant infections in a Chicago teaching hospital: implications for antibiotic stewardship. Clin Infect Dis. 2009;49(8):1175-1184. doi: 10.1086/605630.

4. Carlet J, Collignon P, Goldmann D, et al. Society’s failure to protect a precious resource: antibiotics. Lancet. 2011;378(9788):369-371. doi: 10.1016/S0140-6736(11)60401-7.

5. Wagner B, Filice GA, Drekonja D, et al. Antimicrobial stewardship programs in inpatient hospital settings: a systematic review. Infect Control Hosp Epidemiol. 2014;35(10):1209-1228. doi: 10.1086/678057.

6. Drekonja DM, Filice GA, Greer N, et al. Antimicrobial stewardship programs in outpatient settings: a systematic review. Infect Control Hosp Epidemiol. 2015;36(2):142-152. doi: 10.1017/ice.2014.41.

7. Meeker D, Knight TK, Friedberg MW, et al. Nudging guideline-concordant antibiotic prescribing: a randomized clinical trial. JAMA Intern Med. 2014;174(3):425-431. doi: 10.1001/jamainternmed.2013.14191.

8. Rosenfeld RM, Andes D, Bhattacharyya N, et al. Clinical practice guideline: adult sinusitis. Otolaryngol Head Neck Surg. 2007;137(suppl 3):S1-S31. doi: 10.1016/j.otohns.2007.06.726.

9. Anand VK. Epidemiology and economic impact of rhinosinusitis. Ann Otol Rhinol Laryngol Suppl. 2004;193:3-5.

10. Gill JM, Fleischut P, Haas S, Pellini B, Crawford A, Nash DB. Use of antibiotics for adult upper respiratory infections in outpatient settings: a national ambulatory network study. Fam Med. 2006;38(5):349-354.

11. Jørgensen LC, Friis Christensen S, Cordoba Currea G, Llor C, Bjerrum L. Antibiotic prescribing in patients with acute rhinosinusitis is not in agreement with European recommendations. Scand J Prim Health Care. 2013;31(2):101-105. doi: 10.3109/02813432.2013.788270.

12. Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the Choosing Wisely campaign. JAMA Intern Med. 2015;175(12):1913-1920. doi: 10.1001/jamainternmed.2015.5441.

13. Sharp AL, Klau MH, Keschner D, et al. Low-value care for acute sinusitis encounters: who’s choosing wisely? Am J Manag Care. 2015;21(7):479-485.

14. Fairlie T, Shapiro DJ, Hersh AL, Hicks LA. National trends in visit rates and antibiotic prescribing for adults with acute sinusitis. Arch Intern Med. 2012;172(19):1513-1514. doi: 10.1001/archinternmed.2012.4089.

15. Aring AM, Chan MM. Acute rhinosinusitis in adults. Am Fam Physician. 2011;83(9):1057-1063.

16. Chow AW, Benninger MS, Brook I, et al; Infectious Diseases Society of America. IDSA clinical practice guideline for acute bacterial rhinosinusitis in children and adults. Clin Infect Dis. 2012;54(8):e72-e112. doi: 10.1093/cid/cir1043.

17. Lemiengre MB, van Driel ML, Merenstein D, Young J, De Sutter AI. Antibiotics for clinically diagnosed acute rhinosinusitis in adults. Cochrane Database Syst Rev. 2012;10:CD006089. doi: 10.1002/14651858.CD006089.pub4.

18. Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA. Practical clinical trials for translating research to practice: design and measurement recommendations. Med Care. 2005;43(6):551-557.

19. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi: 10.1136/bmj.38398.500764.8F.

20. Westphal JF, Jehl F, Javelot H, Nonnenmacher C. Enhanced physician adherence to antibiotic use guidelines through increased availability of guidelines at the time of drug ordering in hospital setting. Pharmacoepidemiol Drug Saf. 2011;20(2):162-168. doi: 10.1002/pds.2078.

21. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54. doi: 10.1186/1471-2288-6-54.

22. Munoz-Plaza CE, Parry C, Hahn EE, et al. Integrating qualitative research methods into care improvement efforts within a learning health system: addressing antibiotic overuse. Health Res Policy Syst. 2016;14(1):63. doi: 10.1186/s12961-016-0122-3.

23. Vassar M, Holzmann M. The retrospective chart review: important methodological considerations. J Educ Eval Health Prof. 2013;10:12. doi: 10.3352/jeehp.2013.10.12.

24. Worster A, Bledsoe RD, Cleve P, Fernandes CM, Upadhye S, Eva K. Reassessing the methods of medical record review studies in emergency medicine research. Ann Emerg Med. 2005;45(4):448-451. doi: 10.1016/j.annemergmed.2004.11.021.

25. Levy PS, Lemeshow S. Sampling of Populations: Methods and Applications. 4th ed. Hoboken, NJ: Wiley; 2008.

26. Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267-273. doi: 10.1001/jamainternmed.2013.1589.

27. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2007;14(4):415-423. doi: 10.1197/jamia.M2373.

28. Baker R, Camosso-Stefinovic J, Gillies C, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010;(3):CD005470. doi: 10.1002/14651858.CD005470.pub2.

29. Flottorp SA, Oxman AD, Krause J, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35. doi: 10.1186/1748-5908-8-35.

30. Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA. 2016;315(6):562-570. doi: 10.1001/jama.2016.0275.
Copyright AJMC 2006-2018 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up

Sign In

Not a member? Sign up now!