Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial
Adam L. Sharp, MD, MS; Yi R. Hu, MS; Ernest Shen, PhD; Richard Chen, MD; Ryan P. Radecki, MD, MS; Michael H. Kanter, MD; and Michael K. Gould, MD, MS
Antimicrobial stewardship is key to optimizing patient outcomes and minimizing the emergence of drug-resistant bacterial organisms.1-3 Unfortunately, the goal of reducing inappropriate antibiotic prescribing has remained elusive.4 This is true despite the existence of a variety of strategies to improve antibiotic stewardship, such as pay-for-performance incentives, documentation change strategies, provider education, and social “nudges.”5-7 We currently do not understand why these approaches have variable results across different settings. Research is needed to describe the effect of such interventions in general community practice and to understand variations in effect due to patient, provider, or medical center characteristics in order to inform current efforts to improve care.
Acute sinusitis is a condition that affects 31 million annually8 and costs billions.9 The preponderance of evidence suggests antibiotics are not only ubiquitously overused for acute sinusitis treatment,10-13 but the correct antibiotic is often not chosen.14 Despite agreement regarding overuse, there is no universal recommendation regarding the precise group of patients with acute sinusitis who may benefit from antibiotics.8,15-17 We used the recommendations from our health system, patterned after the Infectious Diseases Society of America (IDSA)’s guideline,16 and a practical study design18 to report the effects of a 2-component intervention designed to improve antibiotic stewardship for acute sinusitis encounters. The components were computerized clinical decision support (CDS) within an electronic health record (EHR)19,20 and provider education.1
Clinical leaders and policy makers need to understand the variable effectiveness of multifaceted interventions, as well as the importance that individual provider characteristics may have on expected results. Our aim was to enhance the understanding of the effects of clinician education and CDS in a real-life clinical setting and, specifically, to assess any differences in effectiveness on providers with varying years of experience or among medical centers.
We performed a pragmatic stepped-wedge cluster randomized study of provider education and CDS for acute sinusitis encounters. The study was performed in outpatient clinics (105 primary care, 21 urgent care) between September 2014 and April 2015 within Kaiser Permanente Southern California (KPSC), a large integrated health system. This system provides healthcare to more than 4 million members representative of the population diversity found in southern California. We used information from the EHR collected during routine care. This study was approved by the KPSC Institutional Review Board.
Each participating primary care and urgent care clinic is clustered within 1 of 6 medical service areas. Each service area was randomly assigned to receive the CDS intervention as part of a staged implementation over the course of 6 months. The total study period was 8 months, allowing for 1 month of pre-data for all areas prior to CDS implementation. CDS was turned on in 1 of the 6 areas each month until all were receiving CDS, then 1 month of post data was collected after all had received the intervention (Figure 1). This staggered implementation allowed for a stepped-wedge analysis.21 This design controls for confounders due to secular trends and strong champion effects.
CDS was based on qualitative interviews with practicing clinicians22 and designed using features shown to be effective in previous trials.19 Automated in the EHR, CDS was initiated at the time of antibiotic prescribing for patients with an acute sinusitis diagnosis. Once triggered, it offered a clear recommendation based on current evidence.16 The alert informed providers that antibiotics were not generally recommended for patients with symptoms of less than 10 days’ duration. Additionally, CDS included options to justify use of antibiotics in the case of prolonged symptoms, “double worsening” of symptoms, or abnormal physical exam findings. CDS could be bypassed by providers, but required a response or cancellation to proceed with the order. In addition, CDS directed providers to an order set to facilitate the prescribing of amoxicillin or amoxicillin clavulanate if they had ordered a nonrecommended antibiotic for acute sinusitis.
Provider education included a recorded online presentation and 2 webinars delivered in the middle of the staggered rollout (December). The intervention was created after previous research showed the need for improved stewardship and no temporal trends in improvement in the years leading up to our study period.13
All acute sinusitis encounters (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] code 461.x) taking place in primary care settings were included. We excluded encounters with specialists, follow-up visits, and patients with chronic diseases compromising their immune response (chronic liver disease, ICD-9-CM code 571; end-stage renal disease, ICD-9-CM code 585.6; congestive heart failure, ICD-9-CM code 428; immune disorders, ICD-9-CM code 279; malignant neoplasms, ICD-9-CM codes 140-165, 170-176, 179-209, and 235-239; and common rheumatologic disorders, ICD-9-CM codes 714, 710, 555.9, and 556). To ensure completeness of data, we limited our sample to Kaiser health plan members.
The primary outcome was prescription of an antibiotic, classified by the generic product identifier. Secondary outcomes included prescription of an antibiotic for diagnoses with similar symptoms as acute sinusitis (acute nasopharyngitis, ICD-9-CM code 460; nonstreptococcal pharyngitis, ICD-9-CM code 462; acute laryngitis, ICD-9-CM code 464; acute laryngopharyngitis, ICD-9 code 465; acute upper respiratory tract infections of other multiple sites, ICD-9-CM code 465.8; acute upper respiratory tract infections not otherwise specified, ICD-9-CM code 465.9; acute bronchitis, ICD-9 code 466; influenza with respiratory manifestations, ICD-9-CM code 487.1; and bronchitis not specified as acute or chronic, ICD-9-CM code 490)7 and concordance with current IDSA16 guidelines.
Concordance estimates were done based on structured chart review from a random sample of 50 charts with some from each of the 6 medical service areas. The structured chart review was performed by a trained member of the research team who followed a standardized protocol to limit bias.23,24 This random sample was selected from encounters in which antibiotics were prescribed post intervention. We used a standardized data abstraction form and manual previously developed to identify guideline-concordant antibiotic use for similar acute sinusitis encounters, which is proven to have high interrater reliability (93% agreement, kappa = .86).13 This previous research provided estimates of concordance rates (32%) for acute sinusitis encounters receiving antibiotics for comparison purposes. We used results from our chart review and applied the sampling fraction to estimate the variance of the proportion of encounters receiving antibiotics concordant with recommendations post intervention. This estimate was calculated via standard formulas routinely used for survey sampling research,25 then applied to the entire sample to construct 95% confidence limits.
We used multiple logistic regression analyses to examine the main effect of CDS on antibiotic prescribing. To account for the staggered implementation of CDS, a dichotomous variable was created to indicate whether the encounter was pre- or post CDS. Another indicator variable was included to assess the impact of education amidst the rollout of CDS. A facility variable was included to identify differences in effect among the service areas. The analysis adjusted for variables that were statistically associated (P <.05) with receipt of antibiotics.
The final model used the provider as a random intercept, the education indicator as a fixed effect, and indicator variables modeled as fixed effects to assess the separate effects of CDS and provider education.
Secondary analyses examined the interaction between CDS and provider groups of varying years of experience. We categorized the years of provider experience into 4 groups (<3 years, 3-9 years, 10-20 years, and >20 years), using the largest group of providers (10-20 years) as the reference for comparison. We similarly examined the interaction between CDS and each of the medical service areas. Within these groups of providers and service areas we also compared the odds of receiving antibiotics pre- and post intervention.
We performed additional analyses to understand changes in sinusitis encounter diagnoses compared with similar upper respiratory infections (URIs) and to evaluate if these encounters resulted in changes to antibiotic prescribing. We report the number of acute sinusitis encounters and other similar URIs, as well as the proportion resulting in antibiotics during the study period (September 2014-April 2015) compared with the same months in the 2 prior years, 2012 to 2013 and 2013 to 2014. Similarly, we report these diagnoses and the percent receiving antibiotics stratified pre-post CDA implementation and pre-post provider education during the study period, as well as the type of antibiotic used pre-post CDS for acute sinusitis encounters.
The study sample included 21,949 initial acute sinusitis encounters for adults (Figure 1). A similar number of encounters were included before and after the implementation of CDS (10,491 pre- and 11,458 post intervention), and more patients were seen at primary care clinics (76%) than in urgent care settings (Table 1).
The CDS intervention significantly reduced antibiotic use for acute sinusitis encounters (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87) (Table 2), although it demonstrated only a 2% absolute reduction. Provider education, assessed as a time effect during December, showed significant decreases in antibiotic prescribing (aOR, 0.51; 95% CI, 0.46-0.57) (Table 2), but these effects were not sustained over the study period (Figure 2).
The odds of receiving antibiotics increased with additional years of provider experience (Table 2), although there was not a statistically significant interaction between years of provider experience and CDS (P = .19) (Table 3). There were significant differences in the effect of CDS and antibiotic prescribing based on the interaction with each medical service area (P <.001).
The overall number of encounters with acute sinusitis diagnoses decreased compared with the same temporal period in the 2 prior years and represented a smaller proportion of acute URI diagnoses (8.1% vs 11.7% and 11.3%). The proportion of acute sinusitis encounters, compared with all similar URI diagnoses, decreased in the posteducation (7.5% vs 9.3%) and the post-CDS (7.1% vs 9.2%) periods compared with pre-intervention. The overall proportion of acute URI encounters receiving antibiotics decreased during the study period (27.1%) compared with prior years (31.2% and 30.7%), but this does not appear to be attributable to the intervention. Pre-post stratification showed there may have been some diagnosis shifting where antibiotics were still prescribed, as CDS (3.3% increase) and education (2% increase) showed a small increase in antibiotic prescribing for other URIs before and after implementation during the study period (eAppendices 1 and 2 [eAppendices available at ajmc.com]). The types of antibiotics used pre-post intervention did not differ significantly (eAppendix 3).
Lastly, a structured review of acute sinusitis encounters for patients who received antibiotics found that 46% (95% CI, 32%-60%) were guideline concordant. This was a 14% absolute improvement from a structured review of acute sinusitis encounters from the same primary care clinics in the pre-implementation period (32%; 95% CI, 19%-45%).13
We performed an effectiveness study of the ability of CDS and provider education to decrease the use of antibiotics for acute sinusitis encounters. Adjusted comparisons showed a 22% improvement in the odds of prescribing antibiotics after the intervention, but the absolute reduction was small (2%). After CDS and education, the number of encounters with an acute sinusitis diagnosis decreased substantially. Lastly, we found that CDS had variable effectiveness based on the medical service area.
This study makes several important contributions to the existing body of literature describing the effectiveness of CDS. First, we confirmed that CDS is effective in real-world clinical settings, but showed that the magnitude of benefit may be less than that observed in highly controlled studies of efficacy.26
Second, we observed a pattern of diagnosis-shifting as a result of education and CDS implementation. This may be an example of documented CDS “workarounds,” and future studies and quality-improvement efforts should account for changes in diagnosis patterns.27 It is possible that providers substituted a diagnosis other than acute sinusitis once they were made aware that antibiotics were not indicated for most patients with sinusitis. Understanding how CDS modifies diagnosis patterns warrants future investigation, although, based on our results, these changes did not result in an overall increase in the use of antibiotics for all URI diagnoses.
Third, we observed increasing odds of antibiotic prescribing correlated with increasing years of provider experience, but this did not result in varying CDS impact. CDS did vary in effectiveness, however, based on the medical service area of the encounter. This may be a result of the culture and context within particular settings that promotes success. This finding supports other reports that have shown these less tangible factors to be an important aspect of the success of interventions.28,29 What is clear is that clinical leaders, policy makers, and researchers should account for the variable effectiveness of CDS depending on the local medical center characteristics.
Despite overall results demonstrating improvements in antibiotic stewardship and guideline-concordant prescribing, we observed a diminution in effect over time. Much of the overall effectiveness of the 2-component intervention may be attributed to the acute drop in prescribing in December associated with provider education, rather than the CDS intervention. This temporal improvement was not sustained, and by the end of the study period rebounded to near the prior baseline. This reinforces that sustained improvement in clinical practice is difficult to achieve.
Our study confirms results from recent efficacy trials showing the benefit of “accountable justification”30 with automated CDS during outpatient encounters, although in a slightly different approach. The limited effectiveness of our results may be partially due to limited “accountability,” as our CDS required a clicked response, but did not require the provider to document in the medical record the reason for use of antibiotics. Additionally, we looked only at acute sinusitis encounters, a condition for which antibiotics are generally not indicated, except under special circumstances, instead of viral diagnoses for which antibiotics are always contraindicated. Based on our relatively large aOR and comparatively small absolute effect of CDS, we hypothesize that patients with shorter symptoms or less severe symptoms were those changed to a different diagnosis and not given antibiotics. This is also based on our chart review, which showed improvements in guideline-concordant use of antibiotics. This may explain the increase in other similar URI diagnoses without a rise in antibiotics for those conditions. Therefore, it is reasonable to predict that the effect of our intervention on overall use of antibiotics may be underestimated by the absolute 2% decrease in the proportion of encounters receiving antibiotics for acute sinusitis.
Provider education and integrated CDS reduced antibiotic use for acute sinusitis encounters and substantially changed patterns of diagnosis for acute URIs. The benefits of education were brief and did not persist through the study period, and CDS effectiveness varied by medical center.
The authors thank Ellen J. Rippberger and Lorena Perez-Reynoso for their administrative assistance and organization throughout this study. They also recognize Jason Doctor, PhD, and Daniella Meeker, PhD, for reviewing and providing feedback about the manuscript.