Ryan P. Radecki, MD, MS

 Patients with chest pain and concern for potential coronary ischemia are frequently referred to the emergency department (ED), resulting in substantial resource utilization and cost. The objective of this study was to implement a protocol for urgent care center (UCC) evaluation of potential acute coronary syndrome (ACS) and describe its performance.

 This is a descriptive, retrospective review of consecutive cases included in a protocol for UCC evaluation of ACS.

 Consecutive patient encounters from 4 urgent care facilities of our regional integrated health system were reviewed from a period spanning 4.5 months of the 2017 calendar year. The primary outcome was avoidance of an ED visit within 30 days of the index visit, and the primary safety outcome was serious adverse events (AEs) occurring in the UCC setting.

 There were 802 patients evaluated, with a median age of 55 years, and 58% were female. Seventy-three (9.1%) patients were referred to the ED or hospitalized for any reason at the index visit, 10 (1.2%) of whom were ultimately diagnosed with ACS. Within 30 days, 56 (7.7%) of the remaining 729 patients had ED visits or hospitalization for any reason, 2 (0.2%) of whom received a diagnosis of ACS. Overall, 673 (83.9%) patients were managed without any ED visit. No serious AEs were recorded.

 Our initial pilot data demonstrate the feasibility of an outpatient UCC evaluation for ACS without refuting the underlying premise of safety.

Adverse events (AEs) during emergency department (ED) evaluation for rule-out acute coronary syndrome (ACS) are exceedingly rare, suggesting that the assumed necessity of ED resources may be questioned. We designed and implemented a formal protocol based on the HEART (History, ECG [electrocardiogram], Age, Risk factors, and Troponin) score and point-of-care troponin testing for evaluation of potential ACS in our outpatient urgent care center (UCC) settings. In a pilot evaluation of 802 consecutive low-risk patients, 84% avoided any ED evaluation within 30 days and none suffered serious AEs.

Existing evidence suggests that most patients undergoing evaluation for ACS are at low risk for AEs.

Risk stratification and point-of-care biomarker testing is feasible in an outpatient UCC setting.

Most patients reviewed in this cohort safely avoided evaluation in our EDs, representing savings to our healthcare organization and of patients’ out-of-pocket costs.

Chest pain is among the most common presenting complaints to the emergency department (ED), estimated to account for 5% to 7% of all visits in the United States.

 The total annualized cost of these visits represents a substantial financial burden, approaching $7 billion, the largest cumulative cost of any ED diagnosis.

 However, the benefit and value of routine referral to an ED setting for evaluation of chest pain, along with its associated higher healthcare costs, are unclear. The preponderance of evidence suggests low rates of life-threatening arrhythmia or serious events in most patients being evaluated for chest pain in the ED or during inpatient hospitalization.

 In particular, these data support the practice standards in which patients with chest pain who are at low risk for acute coronary syndrome (ACS) do not require continuous telemetry during evaluation.

 The logical extension of these data and recommendations further suggests that the infrastructure associated with an acute care facility is not routinely necessary to evaluate patients at low risk for ACS.

The ED at Kaiser Permanente Northwest (KPNW) currently implements a protocol for the rapid evaluation and discharge of low-risk chest pain based on the History, ECG (electrocardiogram), Age, Risk factors, and Troponin (HEART) score.

 Based on positive local and systemwide experience with the HEART score,

 elements of this ED protocol were adapted to our outpatient setting, starting with our urgent care centers (UCCs). Patients meeting low-risk criteria by HEART score were able to receive complete evaluation in the UCC, and health plan members calling the nurse advice line were explicitly directed to UCCs rather than the ED. Implementation of this protocol was followed by ongoing institutional quality assurance evaluation. Our goals for evaluation of our pilot phase were to measure downstream outcomes and resource utilization and detect adverse events (AEs).

This descriptive study details the results of the pilot phase of our UCC protocol. We conducted a retrospective cohort review of patients captured by our UCC testing protocol during a 4.5-month period between July 2017 and November 2017. This study was reviewed by the KPNW Institutional Review Board and determined to be exempt from oversight as relating to continuous quality improvement.

Our regional integrated health system has 2 inpatient hospital acute care facilities, 4 urgent care facilities, and approximately 600,000 members. This pilot phase involved implementation of the outpatient protocol at our UCCs. These facilities lack telemetry monitoring and are staffed by primary and urgent care physicians, physician assistants, and nurse practitioners. Each facility is capable of recording and interpreting an ECG, is equipped with a small laboratory, and transfers patients to a higher level of care when deemed medically necessary.

Patients presenting with symptoms attributable to potential ACS were deemed appropriate for entry into the UCC protocol at one of our urgent care facilities if they met the following criteria: (1) clinically well on subjective assessment, (2) onset of most recent episode of symptoms associated with potential ACS greater than 4 hours prior to presentation, (3) absence of acute ischemia on ECG, and (4) a HEART score prior to obtaining a troponin test result of 3 or lower. Typical patients who would not be considered “clinically well,” and thus would be referred immediately to the ED, included those with active chest pain, hypoxia, clinically important arrhythmias, or dyspnea at rest. A pretroponin HEART score was used under the assumption that a patient with a score of 0 for their troponin would then be eligible for discharge directly from the UCC setting. Patients not meeting these criteria were to be referred to the ED.

When it was necessary to perform a troponin test, the UCC staff utilized the local laboratory’s iStat device with a cTnI (cardiac troponin I) cartridge (Abbott Diagnostics; Abbott Park, Illinois); this assay is not considered to be highly sensitive. The threshold for a clinically positive result was considered to be greater than or equal to 0.033 ng/mL.

All other contemporaneous testing was ordered at the discretion of the treating provider, as was subsequent referral to the ED.

Patients entering this protocol at a UCC in our system were retrospectively identified via electronic health record (EHR) search. All patients for whom a UCC troponin measurement was collected and successfully evaluated were included in our review.

Information systems analysts provided authors with EHR data based on the defined selection criteria. Authors were provided with medical record numbers of patients, as well as the date and time of any subsequent ED visits. Authors were also provided with cardiology staff interpretations of the ECGs, which were then classified by 1 author (R.P.R.) as “normal,” “nonspecific repolarization disturbance,” or “significant ST deviation” per HEART score definitions. Aggregate data on the presence of specific comorbid disease were collated from the structured problem list data recorded in the EHR, which was further used to determine HEART score contributions. One author (B.L.E.-B.) performed individual review of all patients with subsequent ED visits in structured format. These subsequent ED presentations were identified either (1) by encounters at 1 of the 2 acute care hospitals in our integrated health system or (2) via the regional Emergency Department Information Exchange (Collective Medical Technologies, Inc; Draper, Utah). Information collated from individual review included ED chief complaint, primary discharge diagnoses, results of troponin testing, and whether the diagnoses constituted ACS.

The primary outcome of interest was reduction in ED utilization, measured as any visit to an ED within 30 days of the index visit. These visits were stratified as being within 6 hours of or greater than 6 hours since presentation for the index visit. This dichotomy was created with the intention of capturing transfers directly from the UCC at the index visit in the former, separately from independent presentations in the latter. Secondary outcome measures included cardiac and noncardiac diagnoses assigned at subsequent ED visits or hospitalization. The primary safety outcome of interest was any reported serious event during UCC evaluation of potential ACS. This safety end point was defined as clinical deterioration while present in the UCC, including worsening chest pain symptoms, development of malignant cardiac arrhythmia, receipt of advanced cardiac life support, defibrillation, or death.

During the review period between July 5, 2017, and November 20, 2017, 802 patients were evaluated at a UCC for possible ACS and received point-of-care troponin testing. All patients were included in our analysis. The median age of patients tested was 55 years, with an interquartile range of 45 to 66 years. The gender distribution was 58% female. By race, our population was 68% white, 8% Hispanic, 6% black, 5% Pacific Islander, 1% Asian, and 12% multiple races or unknown. Members of the Kaiser Permanente health plan made up 90% of our sample, with 66% commercial insurance purchasers and 24% Medicare product purchasers. The remaining 10% held out-of-network health insurance or were documented as having an unknown insurance status. No Medicaid patients were included in our study population to our knowledge.

The most common comorbidities in our population were obesity, smoking, and hypertension. The distribution of these features in our population is shown in 

. Most patients had normal or nonspecific ECG findings. Individual HEART scores for each patient could not be calculated retrospectively, specifically due to inability to score the History element. An overview of the remainder of the elements of HEART score available at presentation are shown in 

Of these 802 patients, 73 (9.1%) were referred to or evaluated in the ED within 6 hours of the index visit. Of these, 24 had positive iStat troponin tests in the UCC and 10 ultimately received a diagnosis of ACS following further evaluation. All cases of ACS were type 1 non—ST-segment elevation myocardial infarction (NSTEMI). Examples of non-ACS causes for elevated troponin included tachycardia-related demand ischemia, congestive heart failure, sepsis, and pulmonary embolism. These subsequent primary diagnoses are included in 

. The 1 inpatient death in this population resulted from complications following coronary artery bypass grafting (CABG).

Of the remaining 729 patients, 56 (7.6%) were evaluated in the ED within 30 days of a negative UCC evaluation, including assessment for ACS. Two (0.2%) of these patients subsequently had ACS diagnosed. One patient was a 71-year-old male who was evaluated for atypical chest pain and had a negative initial evaluation in the UCC. Thirteen days later, the patient was evaluated for chest pain in the ED, and a quantitative troponin level measured 0.07 ng/mL. He subsequently underwent diagnostic coronary angiography and was referred for CABG. The second patient was a 68-year-old female with chest tightness and body aches who received a negative UCC evaluation and a diagnosis of acute bronchospasm. Two days later, the patient was admitted to the hospital for febrile respiratory illness and contemporaneously found to have a troponin level of 1.83 ng/mL. A nuclear stress test was abnormal, and the patient opted for medical management of presumed ACS. The other primary diagnoses from the subsequent encounters are included in 

. No deaths occurred from any cause. No safety events were reported or identified on review.

We performed a retrospective analysis of a pilot UCC outpatient protocol for the unmonitored evaluation of patients presenting with potential ACS. To our knowledge, this represents the first formal review of such a protocol. Triage protocols for point-of-care or protocolized testing of patients presenting with potential ACS are common as interventions targeting patient throughput, but any unmonitored waiting room time is not by explicit design.

 This unmonitored empiric testing occurs as a pragmatic response to inadequate resource availability. Similar logistics are encountered at our institution, and our novel intervention aims to relieve crowding burdens on our EDs while retaining standards of quality and safety. This also has secondary effects on avoidable resource utilization for the health system and of both timeliness and out-of-pocket costs for patients.

No serious events were detected or reported upon review. This is not unexpected, however, considering that the rate of serious events in low-risk patients hospitalized for the evaluation of chest pain has been reported to be as low as 0.18% by Weinstock et al.

 In their study, of the cases reviewed, none of 7266 patients with normal ECGs and normal vital signs sustained cardiac AEs unless triggered by iatrogenic intervention. This study was not powered or intended to validate the prognostic value of the HEART score, but our observed 1.5% prevalence of ACS within 30 days is similar to that expected for those in a chest pain cohort at low risk for ACS.

In the 10 patients with ACS whose evaluation was initiated at the UCC, all had NSTEMI and underwent early invasive strategies for management. An initial evaluation in UCC introduces delays to definitive diagnosis, including potential harms. No specific harms were identified in our limited cohort.

Continuous quality improvement review for protocol violations was part of this effort, as evaluation of non—low-risk patients increases the likelihood of AEs. For example, if a patient with active chest pain were incorrectly referred to the UCC, this could result in clinically important delays in care if the patient in question were suffering a STEMI. Although this did not occur during our pilot, we did incidentally encounter protocol violations during medical record review, most involving troponin testing in patients with HEART scores greater than 3. These findings inform future individual provider education and quality improvement efforts.

Overall, 673 of 802 (83.9%) patients were successfully managed in the UCC setting with point-of-care troponin testing without ED referral or 30-day recidivism. Prior to implementation of this protocol, all patients in the UCC who were judged to require evaluation for potential ACS would have been transferred via emergency medical services to the ED for evaluation. No reliable data quantifying the cost reduction of this initiative could be obtained, but it is reasonable to suggest that redirecting even a small portion of our national ED burden to outpatient settings might represent a significant reduction in healthcare expenditures.

Our study has many limitations. Although the roll-out of this protocol was closely followed by key administrative leaders, no formal process for submitting safety events or concerns was present. Our integrated health system and EHR typically capture documentation relating to serious events and emergency response to UCCs, but it is still possible that an undocumented safety event was missed. Although 90% of our study population were members of our health plan with reliable 30-day follow-up data, undetected events may have occurred in the remaining 10% whose insurance status was out-of-network or unknown. Given the frequency of expected events overall, the risk of missed adverse outcomes is likely low.

Clinicians had complete individual discretion to include patients in the protocol or to refer them directly to the ED, and this subjectivity at entry introduces a selection bias. HEART score elements abstracted from the medical record may be imprecise due to their dependence on the accuracy of the problem list in structured data. Additional study is necessary to fully quantify a reduction in ED evaluations, as some of the UCC troponin testing may represent “indication creep” resulting from its availability, rather than actual avoided ED referrals. Furthermore, this indication creep may also contribute to the apparent safety profile of this protocol, as the availability of the test may encourage an excess of evaluations in a very low-risk population. Many cases individually reviewed were referred to the ED for further evaluation of other diagnoses, as evidenced by the diversity of final diagnoses in Table 3, suggesting that ACS was not always the most relevant concern. Finally, our results are not generalizable to a setting in which patients with chest pain are not routinely discharged from the ED, as this would likely preclude patients being discharged directly from an UCC setting. Furthermore, it is unclear how these protocols would integrate into a setting in which routine cardiac stress, functional, or anatomic testing occurs in low-risk patients presenting with potential ACS. In our health system, urgent follow-up for stress testing is rare and typically left to the discretion of our primary care providers per routine.

Qualitatively, the implementation of this protocol has been acceptable to clinicians practicing in the UCCs. Initial concerns regarding its roll-out stemmed primarily from concerns regarding lack of monitoring and clinical supervision of patients undergoing evaluation. These were addressed initially with education and subsequently by feedback regarding the ongoing safety monitoring relating to this pilot. Future steps include roll-out to additional primary care locations, along with potentially raising the HEART score threshold for UCC to 5.

We successfully implemented a pilot protocol for UCC evaluation of ACS. Our early results do not refute the underlying premise of safety. Substantial potential healthcare system cost savings may result from moving the evaluation of patients at low risk for ACS to an outpatient, nonhospital venue.

 Kaiser Permanente Northwest (RPR, KFF, TSE, CPH, TEG, HMP, AJM, NRH, BLE-B), Portland, OR.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (RPR, KFF, CPH, HMP, AJM, NRH, BLE-B); acquisition of data (KFF, TSE, TEG, HMP, BLE-B); analysis and interpretation of data (RPR, TEG, AJM, NRH, BLE-B); drafting of the manuscript (RPR, KFF); critical revision of the manuscript for important intellectual content (RPR, BLE-B); statistical analysis (RPR, KFF, CPH, TEG); provision of patients or study materials (TSE); administrative, technical, or logistic support (TSE, CPH, HMP, AJM, NRH, BLE-B); project management support (NRH); and supervision (TSE, CPH, HMP, BLE-B).

 Ryan P. Radecki, MD, MS, Kaiser Permanente Northwest, 500 N Multnomah Ave, Portland, OR 97232. Email: ryan.p.radecki@kp.org.

1. Niska R, Bhuiya F, Xu J. National Hospital Ambulatory Medical Care Survey: 2007 emergency department summary. 

2. Slutzman JE, Wilson M, Cutler D, Schuur JD. Costs attributable to emergency department care by diagnosis. Presented at: American College of Emergency Physicians Scientific Assembly; October 26-27, 2015; Boston, MA. Abstract 188. annemergmed.com/article/S0196-0644(15)00810-0/fulltext.

3. Hollander JE, Sites FD, Pollack CV Jr, Shofer FS. Lack of utility of telemetry monitoring for identification of cardiac death and life-threatening ventricular dysrhythmias in low-risk patients with chest pain. 

4. Singer AJ, Visram F, Shembekar A, Khwaja M, Viccellio A. Telemetry monitoring during transport of low-risk chest pain patients from the emergency department: is it necessary? 

. 2005;12(10):965-969. doi: 10.1197/j.aem.2005.06.007.

Articles

A formal protocol for urgent care center evaluation of potential acute coronary syndrome safely precluded emergency department visits among 84% of those eligible.

Pilot of Urgent Care Center Evaluation for Acute Coronary Syndrome

 Antibiotic stewardship is key to optimizing patient outcomes and affordable care. The study objective was to examine the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis among providers of varying experience.

A stepped-wedge cluster randomized intervention to evaluate antibiotic use for acute sinusitis encounters at 126 Kaiser Permanente Southern California clinics between September 2014 and April 2015.

 The primary outcome was receipt of an antibiotic prescription. Multivariate analysis adjusted for patient, provider, and medical center characteristics. Secondary analyses described sinusitis and other common upper respiratory infection (URI) diagnoses and antibiotic use during the study period compared with prior years. Chart review of a random sample reported the proportion of encounters receiving guideline-concordant antibiotics. 

Analysis of 21,949 encounters (10,491 pre- and 11,458 post intervention) showed CDS reduced the use of antibiotics (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87), although the pre-post absolute difference was small (85.9% vs 83.9%, respectively). Education had a large initial effect (AOR, 0.51; 95% CI, 0.46-0.57), which did not persist. Increasing years of provider experience raised the rates of antibiotic prescribing, but did not have a significant interaction with CDS (

= .19). The effect of CDS varied by medical center (

<.001). In addition, sinusitis diagnoses decreased post intervention, with no overall increase in antibiotic prescribing for URI diagnoses. Lastly, guideline-concordant antibiotic use increased by 14%. 

Provider education and CDS improved antibiotic stewardship and changed diagnosis patterns. The benefits of education were brief, and CDS effectiveness varied by medical center. 

It is vital to understand the effectiveness of interventions designed to improve patient care in order to disseminate and replicate successful initiatives and avoid fruitless care improvement efforts. Our study of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis offers the following to inform health systems and future research: 

CDS integrated in an electronic health record can discourage inappropriate antibiotic prescribing and will likely change diagnostic patterns. 

Provider education is unlikely to have a sustained effect on antibiotic prescribing.

The effectiveness of education and CDS will vary based on inherent differences among locations.

Antimicrobial stewardship is key to optimizing patient outcomes and minimizing the emergence of drug-resistant bacterial organisms.

 Unfortunately, the goal of reducing inappropriate antibiotic prescribing has remained elusive.

 This is true despite the existence of a variety of strategies to improve antibiotic stewardship, such as pay-for-performance incentives, documentation change strategies, provider education, and social “nudges.”

 We currently do not understand why these approaches have variable results across different settings. Research is needed to describe the effect of such interventions in general community practice and to understand variations in effect due to patient, provider, or medical center characteristics in order to inform current efforts to improve care. 

Acute sinusitis is a condition that affects 31 million annually

 The preponderance of evidence suggests antibiotics are not only ubiquitously overused for acute sinusitis treatment,

 but the correct antibiotic is often not chosen.

 Despite agreement regarding overuse, there is no universal recommendation regarding the precise group of patients with acute sinusitis who may benefit from antibiotics.

 We used the recommendations from our health system, patterned after the Infectious Diseases Society of America (IDSA)’s guideline,

 to report the effects of a 2-component intervention designed to improve antibiotic stewardship for acute sinusitis encounters. The components were computerized clinical decision support (CDS) within an electronic health record (EHR)

Clinical leaders and policy makers need to understand the variable effectiveness of multifaceted interventions, as well as the importance that individual provider characteristics may have on expected results. Our aim was to enhance the understanding of the effects of clinician education and CDS in a real-life clinical setting and, specifically, to assess any differences in effectiveness on providers with varying years of experience or among medical centers. 

We performed a pragmatic stepped-wedge cluster randomized study of provider education and CDS for acute sinusitis encounters. The study was performed in outpatient clinics (105 primary care, 21 urgent care) between September 2014 and April 2015 within Kaiser Permanente Southern California (KPSC), a large integrated health system. This system provides healthcare to more than 4 million members representative of the population diversity found in southern California. We used information from the EHR collected during routine care. This study was approved by the KPSC Institutional Review Board.

Each participating primary care and urgent care clinic is clustered within 1 of 6 medical service areas. Each service area was randomly assigned to receive the CDS intervention as part of a staged implementation over the course of 6 months. The total study period was 8 months, allowing for 1 month of pre-data for all areas prior to CDS implementation. CDS was turned on in 1 of the 6 areas each month until all were receiving CDS, then 1 month of post data was collected after all had received the intervention (

). This staggered implementation allowed for a stepped-wedge analysis.

 This design controls for confounders due to secular trends and strong champion effects. 

CDS was based on qualitative interviews with practicing clinicians

 and designed using features shown to be effective in previous trials.

 Automated in the EHR, CDS was initiated at the time of antibiotic prescribing for patients with an acute sinusitis diagnosis. Once triggered, it offered a clear recommendation based on current evidence.

 The alert informed providers that antibiotics were not generally recommended for patients with symptoms of less than 10 days’ duration. Additionally, CDS included options to justify use of antibiotics in the case of prolonged symptoms, “double worsening” of symptoms, or abnormal physical exam findings. CDS could be bypassed by providers, but required a response or cancellation to proceed with the order. In addition, CDS directed providers to an order set to facilitate the prescribing of amoxicillin or amoxicillin clavulanate if they had ordered a nonrecommended antibiotic for acute sinusitis. 

Provider education included a recorded online presentation and 2 webinars delivered in the middle of the staggered rollout (December). The intervention was created after previous research showed the need for improved stewardship and no temporal trends in improvement in the years leading up to our study period.

International Classification of Diseases, Ninth Revision, Clinical Modification

] code 461.x) taking place in primary care settings were included. We excluded encounters with specialists, follow-up visits, and patients with chronic diseases compromising their immune response (chronic liver disease, 

 codes 140-165, 170-176, 179-209, and 235-239; and common rheumatologic disorders, 

 codes 714, 710, 555.9, and 556). To ensure completeness of data, we limited our sample to Kaiser health plan members. 

The primary outcome was prescription of an antibiotic, classified by the generic product identifier. Secondary outcomes included prescription of an antibiotic for diagnoses with similar symptoms as acute sinusitis (acute nasopharyngitis, 

 code 465; acute upper respiratory tract infections of other multiple sites, 

 code 465.8; acute upper respiratory tract infections not otherwise specified, 

 code 466; influenza with respiratory manifestations, 

 code 487.1; and bronchitis not specified as acute or chronic, 

Concordance estimates were done based on structured chart review from a random sample of 50 charts with some from each of the 6 medical service areas. The structured chart review was performed by a trained member of the research team who followed a standardized protocol to limit bias.

 This random sample was selected from encounters in which antibiotics were prescribed post intervention. We used a standardized data abstraction form and manual previously developed to identify guideline-concordant antibiotic use for similar acute sinusitis encounters, which is proven to have high interrater reliability (93% agreement, kappa = .86).

 This previous research provided estimates of concordance rates (32%) for acute sinusitis encounters receiving antibiotics for comparison purposes. We used results from our chart review and applied the sampling fraction to estimate the variance of the proportion of encounters receiving antibiotics concordant with recommendations post intervention. This estimate was calculated via standard formulas routinely used for survey sampling research,

 then applied to the entire sample to construct 95% confidence limits. 

We used multiple logistic regression analyses to examine the main effect of CDS on antibiotic prescribing. To account for the staggered implementation of CDS, a dichotomous variable was created to indicate whether the encounter was pre- or post CDS. Another indicator variable was included to assess the impact of education amidst the rollout of CDS. A facility variable was included to identify differences in effect among the service areas. The analysis adjusted for variables that were statistically associated (

The final model used the provider as a random intercept, the education indicator as a fixed effect, and indicator variables modeled as fixed effects to assess the separate effects of CDS and provider education. 

Secondary analyses examined the interaction between CDS and provider groups of varying years of experience. We categorized the years of provider experience into 4 groups (<3 years, 3-9 years, 10-20 years, and >20 years), using the largest group of providers (10-20 years) as the reference for comparison. We similarly examined the interaction between CDS and each of the medical service areas. Within these groups of providers and service areas we also compared the odds of receiving antibiotics pre- and post intervention. 

We performed additional analyses to understand changes in sinusitis encounter diagnoses compared with similar upper respiratory infections (URIs) and to evaluate if these encounters resulted in changes to antibiotic prescribing. We report the number of acute sinusitis encounters and other similar URIs, as well as the proportion resulting in antibiotics during the study period (September 2014-April 2015) compared with the same months in the 2 prior years, 2012 to 2013 and 2013 to 2014. Similarly, we report these diagnoses and the percent receiving antibiotics stratified pre-post CDA implementation and pre-post provider education during the study period, as well as the type of antibiotic used pre-post CDS for acute sinusitis encounters. 

The study sample included 21,949 initial acute sinusitis encounters for adults (Figure 1). A similar number of encounters were included before and after the implementation of CDS (10,491 pre- and 11,458 post intervention), and more patients were seen at primary care clinics (76%) than in urgent care settings (

The CDS intervention significantly reduced antibiotic use for acute sinusitis encounters (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87) (

), although it demonstrated only a 2% absolute reduction. Provider education, assessed as a time effect during December, showed significant decreases in antibiotic prescribing (aOR, 0.51; 95% CI, 0.46-0.57) (Table 2), but these effects were not sustained over the study period (

The odds of receiving antibiotics increased with additional years of provider experience (Table 2), although there was not a statistically significant interaction between years of provider experience and CDS (

). There were significant differences in the effect of CDS and antibiotic prescribing based on the interaction with each medical service area (

The overall number of encounters with acute sinusitis diagnoses decreased compared with the same temporal period in the 2 prior years and represented a smaller proportion of acute URI diagnoses (8.1% vs 11.7% and 11.3%). The proportion of acute sinusitis encounters, compared with all similar URI diagnoses, decreased in the posteducation (7.5% vs 9.3%) and the post-CDS (7.1% vs 9.2%) periods compared with pre-intervention. The overall proportion of acute URI encounters receiving antibiotics decreased during the study period (27.1%) compared with prior years (31.2% and 30.7%), but this does not appear to be attributable to the intervention. Pre-post stratification showed there may have been some diagnosis shifting where antibiotics were still prescribed, as CDS (3.3% increase) and education (2% increase) showed a small increase in antibiotic prescribing for other URIs before and after implementation during the study period (

]). The types of antibiotics used pre-post intervention did not differ significantly (

Lastly, a structured review of acute sinusitis encounters for patients who received antibiotics found that 46% (95% CI, 32%-60%) were guideline concordant. This was a 14% absolute improvement from a structured review of acute sinusitis encounters from the same primary care clinics in the pre-implementation period (32%; 95% CI, 19%-45%).

We performed an effectiveness study of the ability of CDS and provider education to decrease the use of antibiotics for acute sinusitis encounters. Adjusted comparisons showed a 22% improvement in the odds of prescribing antibiotics after the intervention, but the absolute reduction was small (2%). After CDS and education, the number of encounters with an acute sinusitis diagnosis decreased substantially. Lastly, we found that CDS had variable effectiveness based on the medical service area.

This study makes several important contributions to the existing body of literature describing the effectiveness of CDS. First, we confirmed that CDS is effective in real-world clinical settings, but showed that the magnitude of benefit may be less than that observed in highly controlled studies of efficacy.

Second, we observed a pattern of diagnosis-shifting as a result of education and CDS implementation. This may be an example of documented CDS “workarounds,” and future studies and quality-improvement efforts should account for changes in diagnosis patterns.

 It is possible that providers substituted a diagnosis other than acute sinusitis once they were made aware that antibiotics were not indicated for most patients with sinusitis. Understanding how CDS modifies diagnosis patterns warrants future investigation, although, based on our results, these changes did 

 result in an overall increase in the use of antibiotics for all URI diagnoses. 

Third, we observed increasing odds of antibiotic prescribing correlated with increasing years of provider experience, but this did not result in varying CDS impact. CDS did vary in effectiveness, however, based on the medical service area of the encounter. This may be a result of the culture and context within particular settings that promotes success. This finding supports other reports that have shown these less tangible factors to be an important aspect of the success of interventions.

 What is clear is that clinical leaders, policy makers, and researchers should account for the variable effectiveness of CDS depending on the local medical center characteristics. 

Despite overall results demonstrating improvements in antibiotic stewardship and guideline-concordant prescribing, we observed a diminution in effect over time. Much of the overall effectiveness of the 2-component intervention may be attributed to the acute drop in prescribing in December associated with provider education, rather than the CDS intervention. This temporal improvement was not sustained, and by the end of the study period rebounded to near the prior baseline. This reinforces that sustained improvement in clinical practice is difficult to achieve. 

Our study confirms results from recent efficacy trials showing the benefit of “accountable justification”

 with automated CDS during outpatient encounters, although in a slightly different approach. The limited effectiveness of our results may be partially due to limited “accountability,” as our CDS required a clicked response, but did not require the provider to document in the medical record the reason for use of antibiotics. Additionally, we looked only at acute sinusitis encounters, a condition for which antibiotics are generally not indicated, except under special circumstances, instead of viral diagnoses for which antibiotics are always contraindicated. Based on our relatively large aOR and comparatively small absolute effect of CDS, we hypothesize that patients with shorter symptoms or less severe symptoms were those changed to a different diagnosis and not given antibiotics. This is also based on our chart review, which showed improvements in guideline-concordant use of antibiotics. This may explain the increase in other similar URI diagnoses without a rise in antibiotics for those conditions. Therefore, it is reasonable to predict that the effect of our intervention on overall use of antibiotics may be underestimated by the absolute 2% decrease in the proportion of encounters receiving antibiotics for acute sinusitis.

Provider education and integrated CDS reduced antibiotic use for acute sinusitis encounters and substantially changed patterns of diagnosis for acute URIs. The benefits of education were brief and did not persist through the study period, and CDS effectiveness varied by medical center.

The authors thank Ellen J. Rippberger and Lorena Perez-Reynoso for their administrative assistance and organization throughout this study. They also recognize Jason Doctor, PhD, and Daniella Meeker, PhD, for reviewing and providing feedback about the manuscript. 

 Department of Research and Evaluation, Kaiser Permanente Southern California (ALS, YRH, ES, MKG), Pasadena, CA; Department of Emergency Medicine, Los Angeles Medical Center, Kaiser Permanente Southern California (ALS), Los Angeles, CA; Department of Emergency Medicine, San Diego Medical Center, Kaiser Permanente Southern California (RC), San Diego, CA; Department of Emergency Medicine, Kaiser Permanente Northwest (RPR), Portland, OR; Medical Director of Quality and Clinical Analysis, Kaiser Permanente Southern California (MHK), Pasadena, CA.

 Internal funding from the KPSC Care Improvement Research Team (CIRT) supported this project.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

 Concept and design (ALS, ES, RC, MHK); acquisition of data (YRH); analysis and interpretation of data (ALS, YRH, ES, RC, RPR, MHK, MKG); drafting of the manuscript (ALS, ES, RPR); critical revision of the manuscript for important intellectual content (ALS, ES, RPR, MHK, MKG); statistical analysis (YRH, ES); administrative, technical, or logistic support (RC); and supervision (ALS).

 Adam L. Sharp, MD, MS, Kaiser Permanente Southern California, 100 S Los Robles Ave, 2nd Fl, Pasadena, CA 91101. E-mail: adam.l.sharp@kp.org. 

1. Society for Healthcare Epidemiology of America; Infectious Diseases Society of America; Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). 

. 2012;33(4):322-327. doi: 10.1086/665010.

2. Fishman N. Antimicrobial stewardship. 

. 2006;119(6 suppl 1):S53-S61; discussion S62-S70. doi: 10.1016/j.amjmed.2006.04.003.

3. Roberts RR, Hota B, Ahmad I, et al. Hospital and societal costs of antimicrobial-resistant infections in a Chicago teaching hospital: implications for antibiotic stewardship. 

. 2009;49(8):1175-1184. doi: 10.1086/605630.

4. Carlet J, Collignon P, Goldmann D, et al. Society’s failure to protect a precious resource: antibiotics. 

. 2011;378(9788):369-371. doi: 10.1016/S0140-6736(11)60401-7.

The authors assessed the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis. Education and CDS improved antibiotic stewardship and changed diagnosis patterns.

Ryan P. Radecki, MD, MS

Articles

Pilot of Urgent Care Center Evaluation for Acute Coronary Syndrome

Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial