Stephen D. Persell, MD, MPH

To evaluate the effects of a multifaceted quality improvement intervention during 2 time periods on 4 coronary artery disease [CAD] measures in 4 primary care practices. During the first phase, electronic reminders prompted physicians to order indicated medications or record contraindications and refusals (exceptions). In the second phase, physicians also received reports about their performance (including lists of patients not satisfying these measures), and financial incentives were announced.

Adult CAD patients seen within the preceding 18 months were included. The primary outcome was the performance on each measure (proportion of eligible patients satisfying each measure after removing those with exceptions). Secondary outcomes were the proportion with the medication on their medication list, and the proportion with exceptions.

Median performance at baseline was 78.8% for antiplatelet treatment, 85.1% for statin treatment, 77.0% for beta-blocker after myocardial infarction (MI), and 67.1% for angiotensinconverting enzyme inhibitor or angiotensin receptor blocker after MI. Performance improved slightly for 3 measures during the first phase and improved more substantially for all 4 measures during the second phase. For 3 of 4 measures, however, documentation of exceptions increased but not medication prescribing. Most exceptions were judged to be appropriate by peer review.

Physicians responded more to the combination of feedback and financial incentives than they had to electronic reminders alone. High performance was only achieved for 1 of 4 measures and recording of exceptions rather than increases in medication prescribing accounted for most of the observed improvements.

We examined the sequential implementation of commonly used quality improvement techniques to improve outpatient coronary heart disease care in 4 practices.

Physician response to electronic reminders alone to address outpatient coronary heart disease measures was small.

Physicians responded more when the combination of feedback reports and financial incentives was added to reminders.

For 3 of 4 measures, documentation of exceptions increased, but not medication prescribing.

Most physician-recorded exceptions to quality measures were judged to be appropriate by peer review.

Quality improvement techniques that leverage an electronic health record (EHR) have been shown to improve care in many cases.

However, EHR-based quality improvement has not been universally successful, and even in many instances where study results were positive, the magnitude of the improvement was small.

Furthermore, observational data do not suggest that simply having an EHR improves quality in outpatient settings.

 In contrast, we have shown in the UPQUAL study (Utilizing Precision Performance Measurement for Focused Quality Improvement) that interconnected EHR-based tools can improve quality for multiple process of care measures in a large urban, single-site, university-affiliated practice.

 This intervention was designed to improve quality measurement (including capture of contraindications and patient refusals), make point-of-care reminders more accurate, and provide more valid and responsive feedback to clinicians (including lists of patients not receiving essential medications).

In this current study, we applied these principles—improve quality measurement in order to enable more accurate point-of-care reminders and feedback&mdash;to coronary artery disease (CAD) care in 4 suburban primary care group practices (2 family medicine and 2 internal medicine) that belong to the same health system and use the same EHR. We selected CAD care as our study objective because it is a common and important chronic disease and because implementing CAD measures in this setting was more feasible than several other candidate chronic disease and prevention topics. In this health system, point-of-care reminders were implemented first in July 2008 (Phase 1). Starting September to November 2009 (Phase 2), feedback was given to physicians and the medical group publicized to physicians that financial incentives would be tied to performance measures (including the 4 measures studied here). Both the reminders and physician feedback portions of the interventions were planned prior to Phase 1 by the study team. The financial incentives were initiated independently by leadership in the organization that was not directly associated with this study. This sequential implementation provides an opportunity to observe the additional effects of adding the combination of feedback and announcing financial incentives to electronic reminders on measured performance.

We performed this study at 4 primary care practices in the northern suburbs of Chicago, Illinois, that use the same commercial EHR (EpicCare, Epic Systems Corporation, Verona, Wisconsin). Northwestern University’s and Northshore University HealthSystem’s institutional review boards approved the study. All patients eligible for 1 or more quality measures cared for by 33 attending physicians (10 family medicine and 23 internal medicine) and 15 family medicine resident physicians were included. The practices had used the EHR for 5 years before the start of the first intervention examined in the study.

Sequential Implementation of Quality Improvement Techniques

We selected for consideration 4 measures of CAD care quality that were based on national measures: antiplatelet drug and lipid-lowering drug treatment in all patients with CAD, beta-blocker use in patients with prior myocardial infarction (MI), and angiotensin-converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) for diabetes or left ventricular systolic dysfunction.

 Health system clinicians, including cardiologists, discussed and modified the measures for local use, changing the lipid-lowering drug measure to statin treatment in CAD and changing the ACE inhibitor or ARB measure to apply to patients with prior MI only.

Electronic Clinical Decision Support (Phase 1)

Prior to these interventions, there were no other clinical decision support tools in use that addressed these topics. We added electronic point-of-care reminders that appeared during patient encounters when an apparently eligible patient did not have an indicated medication on their current medication list and had no exception recorded. These alerts were minimally intrusive (they did not interrupt clinicians’ work flow and the alert was indicated only by a single yellow highlighted tab that appeared on the left side of the screen when any clinical reminder criteria were present, and physicians had to select this tab to see the individual reminders). These alerts were displayed using existing EpicCare functionality. These electronic reminders included standardized ways to capture patient reasons (eg, refusals) or medical reasons that were exceptions for individual reminders within the reminder system of the EHR. These reminders were implemented in July 2008. We sent physicians educational e-mails with brief training materials to introduce the new alerts and to show how to record patient or medical exceptions.

Implementation of Feedback Reports and Announcement of Incentives (Phase 2)

Starting in September 2009, on a monthly basis, we gave physicians printed reports indicating their overall performance on each of the 4 measures for all their eligible patients and lists of individual patients who appeared to be eligible for an indicated medication but were not receiving it and had no exception recorded.

In October and November of 2009, the medical group leadership announced to physicians that a small portion of their compensation (1.5% of total compensation, which constituted 25% of the incentive-based compensation) would be tied to their performance on quality metrics, including the 4 metrics covered in this study.

We retrospectively calculated the performance for the 4 CAD measures for each month from September 2007 through March 2011. At each time point, patients were eligible for a measure if they had an office visit with a physician from 1 of the 4 included practices during the preceding 18 months and had a qualifying

 International Classification of Diseases, Ninth Revision, Clinical Modification 

code used on their active problem list, past medical history, or as an encounter diagnosis. We used Structured Query Language to retrieve data from an enterprise data warehouse that contains data copied daily from the EHR. For each time point, all patients were classified for each measure for which they were eligible as: a) satisfied, b) did not satisfy but had an exception, or c) did not satisfy and had no documented exception. The primary outcome for each measure was calculated as the number of patients who satisfied the measure divided by the total number of eligible patients excluding those with an exception. As an equation, the primary outcome = number satisfied / [number eligible — number not satisfied with an exception]. We also analyzed separately for each measure the proportion of eligible patients who satisfied the measure (were given the medication) and the proportion of all eligible patients who did not satisfy the measure and had exceptions.

We performed peer review beginning in September 2009 of medical exceptions recorded in the EHR and continued the review process for exceptions entered within the first 10 months of the intervention. One physician reviewed medical records to collect the reason for the exception and additional clinical information needed to judge the validity of the exception. When the clinical reasoning was unclear, the peer reviewer would request clarification from the treating clinician. Two board-certified internists and 1 board-certified family medicine physician met regularly to review the exceptions and judged them as appropriate, inappropriate (including when no contraindication to the medication was evident on physician chart review or cases where clarifying information was requested from the primary care physician but was not provided), or of uncertain appropriateness by consensus. When a consensus was not reached or the appropriateness was uncertain, 1 physician reviewed the medical literature, requested advice from specialists when needed, and the group discussed the case again until consensus was reached. Practice physicians received e-mail or telephone feedback for medical exceptions that were judged to be inappropriate.

We used interrupted time series analysis to examine changes in the primary and secondary outcomes over time for 2 different interventions: before and after July 2008 (the transition between baseline and Phase 1), and before and after September 2009 (the transition between Phase 1 and Phase 2). We calculated the primary and secondary outcomes for each of the performance measures for each month from September 2007 through August 2010. A linear model was fit to each series including a continuous time variable, a dichotomous indicator of the intervention, and the interaction term of time and intervention as covariates. The individual data points used for each time period are depicted in the figures. Next, we determined the autoregressive order of the model residuals by minimizing Akaike’s information criterion.

 Finally, we fit a linear regression model with autoregressive errors (using the appropriate number of autoregressive parameters, if any were necessary) to each series. These fitted models were used to test statistical significance.

 To ensure model validity, we examined several residual diagnostics, the Jarque-Bera and the Shapiro-Wilk tests for normality of residuals, and normal Q-Q and autocorrelation plots.

 Analyses used SAS version 9.2 (SAS Institute Inc, Cary, North Carolina) and R software package version 2.13.1 (R Foundation for Statistical Computing, Vienna, Austria).

The number of patients eligible for the coronary disease measures and their characteristics are provided in 

. The number of eligible patients increased over time from 779 CAD and 218 MI patients in October 2007 to 1099 CAD and 332 MI patients by October 2010. Their characteristics changed little during the 3 years we examined (Table 1).

Median performance during the baseline period was 78.9% for antiplatelet treatment, 85.3% for statin treatment, 77.0% for beta-blocker after MI, and 67.2% for ACE inhibitor or ARB after MI (

). Performance on the antiplatelet measure was increasing significantly during the baseline period. Performance on the other 3 measures did not change significantly during the baseline period (

During Phase 1, overall performance continued to increase for the antiplatelet measure at a rate that was similar to the rate of improvement observed during the baseline period. There were statistically significant, but very small, increases in overall performance for the statin and ACE inhibitor/ARB measures (Table 2 and Figure 1). However, there were no significant increases in the rates of patients given medication for these 2 measures, and there was a decrease in patients given beta-blockers during Phase 1 compared with baseline (Table 2 and Figure 2). Physicians recorded exceptions to these measures during Phase 1 for small percentages of eligible patients (

There were significant increases in measured performance for all 4 measures during Phase 2 compared with Phase 1 (Table 2 and Figure 1). For 3 of the 4 measures, there was an increase in the documentation of exceptions (Table 2 and Figure 3) but not medication prescribing (Figure 2). For the antiplatelet measure there was a significant increase in both medication prescribing and exception documentation (Table 2, Figures 2 and 3).

By March 2011, overall performance was 99.7%, 91.1%, 86.8%, and 78.9% for antiplatelet treatment, statin treatment, beta-blocker after MI, and ACE inhibitor or ARB after MI, respectively.

We performed peer review for 179 medical exceptions. For 13 of these, a physician recorded an exception even though the patient was given the medication (the physician both recorded an exception and ordered the medication). Exceptions of this kind would not be included in the performance calculation, since the numerator criteria were met. Of the remaining 166 exceptions, 145 (87.3%) were judged appropriate on peer review,19 (11.4%) had an inappropriate reason or had no reason found for not prescribing on peer review, and 2 (1.2%) were of uncertain appropriateness (

We used time series analysis to examine the effects of implementing electronic clinician reminders directed at 4 measures of outpatient coronary artery disease care followed later by physician audit and feedback, lists of individual patients not meeting the measures, and the announcement of financial incentives. We made several observations that warrant discussion.

Introducing electronic reminders alone during Phase 1 had very little impact on measured quality. Physicians did interact with the clinical decision support (CDS) system to some degree, as evidenced by recording of exceptions using the CDS system. However, the extent of this usage was modest, and led to very small changes in measured quality. These findings are consistent with older studies of CDS aimed at changing outpatient provider prescribing behavior for coronary artery disease that found small effects, or no effect, from electronic clinician reminders.

After the start of individualized feedback to physicians with lists of patients not satisfying the measures and the announcement that there would be financial incentives based on performance (Phase 2), there were much more prominent changes in performance. During this second intervention period, there was a sharp increase in physicians’ use of exception recording for all 4 measures as well as an increase in patients with antiplatelet medication recorded on their medication lists. As a result of these changes, there were significant increases in measured performance for all 4 measures. This observed change suggests that linking CDS tools to local accountability systems like incentives or providing feedback on performance can lead to greater physician engagement with EHR quality improvement tools than would occur with CDS alone. Prior studies have shown a positive effect of providing audit and feedback to physicians,

 and inconsistent effects of financial incentives to improve quality.

Because performance feedback and the announcement of incentives occurred at nearly the same time in this healthcare system, we cannot distinguish the relative contributions of each to the observed changes. Future studies should examine whether adding financial incentives to performance feedback produces a greater effect on quality than feedback alone. This distinction is important since the long-term sustainability of financial incentives requires that the organization commit some amount of financial resources to prioritizing quality over other behaviors such as clinical volume.

Articles

We examined the impact of electronic reminders followed by performance reports and financial incentives. Physicians responded more to reports and incentives than to reminders alone.

Implementation of EHR-Based Strategies to Improve Outpatient CAD Care

: To determine the effect of standardized outreach on the receipt of preventive services for patients whose physicians record that the patient refused the service.

 Prospective observational study of a quality improvement intervention using a nonrandomly assigned comparator group.

 Patients from a large internal medicine practice with recorded refusals to preventive services were included. A nonclinician care manager mailed plain-language educational brochures, attempted telephone contact, and provided logistical assistance. The primary patient outcome was the time from refusal to first receipt of a refused service (colorectal cancer screening, breast cancer screening [mammography], cervical cancer screening, osteoporosis screening [bone density testing], or pneumococcal vaccination). We compared the time to completion of refused sevices during the period when outreach was performed (February 8, 2008, to November 25, 2008 [outreach cohort]), and during a subsequent period when refusals were recorded but no outreach was performed (November 26, 2008, to December 1, 2009 [nonintervention cohort]), using Cox proportional hazards regression models adjusted for patient characteristics. We recorded the time spent performing outreach.

 In total, 407 patients refused 520 preventive services in the outreach cohort, and 378 patients refused 510 services in the nonintervention cohort. After 6 months of follow-up, 6.1% of the outreach cohort and 4.8% of the nonintervention cohort had received a refused service (adjusted hazard ratio, 1.3; 95% confidence interval, 0.7-2.5). The care manager spent 214 hours performing the outreach.

 Standardized educational outreach was not a promising strategy for improving preventive services use among patients who have refused services recommended by their physician. The amount of time required to perform the outreach was substantial.

	This study examined the effects of standardized outreach on the receipt of preventive services for patients whose physicians record that the patient refused the service. A prospective observational study design included a nonrandomly assigned comparator group.

		This intervention, which included mailed plain-language educational brochures, attempted telephone contact, and assistance with scheduling, led to few patients (6.1%) obtaining the refused service over 6 months.

		The receipt of refused services did not significantly differ between the outreach cohort and the nonintervention cohort.

		The care manager time required to perform this outreach was substantial.

	Interventions that increase healthcare providers’ recommendations to patients to obtain preventive services have been proved to increase the use of these services.

 However, some patients refuse effective preventive services when they are offered.

 How the healthcare delivery system should address these refusals is unclear. Clinicians must respect patient autonomy even when patients make choices to forgo beneficial tests or treatments.

 However, if patients lack adequate information about why a test or treatment is recommended, their voiced refusal may indicate a spurious preference, meaning it is not the choice they would make if they had better information. Concerns have been raised that brief office visits may not provide adequate opportunity for clinicians to educate patients about the host of services that may be of benefit to them.

 The information provided by clinicians during office visits may be difficult for some patients to understand, particularly those with lower literacy or socioeconomic status.

 Patients may hold misconceptions about preventive services that lead them to believe that the services are undesirable, would not benefit them, or may be harmful.

Patients may refuse a medical service if they think that a perceived barrier has not been addressed.

 Standardized educational outreach and care facilitation could address these concerns and lead some patients who initially refused a service to accept it.

	As part of a larger multifaceted quality improvement intervention,

we sought to implement a system of care to ensure that the following goals were met among patients who had refused a recommended preventive service: (1) the patients received clear educational material that provided the essential rationale justifying the need for the preventive service, (2) the information was written at a low-literacy level, (3) some common misconceptions about the service were specifically addressed, and (4) patients had the opportunity to discuss barriers to receiving the service with a nonclinician care manager. This study compares the receipt of refused preventive services among patients receiving care during the period when outreach was conducted at this practice site and during a subsequent period when no outreach was performed.

	We performed this study at a large academic primary care inter-nal medicine practice in Chicago, Illinois, using a commercial electronic health record (EHR) (EpicCare, spring 2007 version; Epic Systems Corporation, Verona, WI). Northwestern University’s institutional review board approved the study. Thirty-nine attending physicians, 1 nurse practitioner, and more than 120 residents worked at the clinic during this time.

	We previously reported the details of the overall quality improvement context within which this study took place.

 Physicians received quarterly reports of the quality of care received by their primary care patients and had point-of-care reminders provided by the clinical decision support system in the EHR. Physicians were encouraged to record preventive services that patients reported receiving from other providers. Physicians were asked to record standardized exceptions that explained why a service that seemed to be indicated was not performed. We suggested that physicians record a medical exception when a service was not recommended because of a contraindication, an intolerance, or a suspected lack of clinical benefit because of a patient’s clinical condition. We encouraged physicians to record a patient refusal when they recommended a test or treatment but the patient actively refused. We advised physicians not to record an exception when there was a transient reason for not performing a service or if the patient intended to obtain the service elsewhere. Because recording of exceptions turned off computerized reminders, physicians were advised that they should not record an exception if they wanted to be reminded to address a preventive care issue again at the patient’s next visit.

	Patients were eligible for inclusion in this study if a physician entered standardized documentation in the EHR that the patient had refused a preventive service that the physician recommended. The preventive services we examined included colorectal cancer screening, breast cancer screening (mammography), cervical cancer screening, osteoporosis screening (bone density testing), and pneumococcal vaccination; the numbers of patients in the practice eligible for each of these services at the start of the study were 7067, 3539, 7462, 2966, and 1816, respectively, approximately 12,000 patients overall.

 Patients with refusals documented between February 8, 2008, and November 25, 2008, constituted the outreach cohort. We stopped performing outreach after 10 months because the ongoing resources that were required did not seem justified. We selected patients who had refusals documented during the period immediately following the intervention, November 26, 2008, to December 1, 2009, as a comparator nonintervention cohort (excluding patients who were already included in the outreach cohort). This group was selected for comparison in a quasiexperimental design to assess whether there was an underlying rate at which patients who refused services but received no other structured intervention were obtaining these services.

 We selected a longer period for the nonintervention cohort to increase the size of the population available and to raise the statistical power for comparisons.

	During the outreach period, we performed an automated search of the EHR each week to identify patients with any new refusals recorded to electronic reminders for the 5 preventive services aforelisted. A nonclinician care manager (EMF) performed the following task: She reviewed the EHR to determine if any specific barriers to obtaining the refused service were documented and to assess if outreach for a preventive service seemed inappropriate. Patients who had substantial active medical or psychosocial stressors described in the EHR or for whom the preventive service seemed medically contraindicated were deemed inappropriate for outreach, and no physician contact was performed. These reasons were reviewed at weekly meetings with physician team members (SDP, NCD, DWB), who helped decide when outreach should not be performed. The care manager then sent e-mails within the EHR to patients’ primary care physicians notifying them that patients would receive outreach unless the physician indicated that outreach should not be performed, mailed patients brief materials that included plain-language educational brochures relevant to each topic, attempted telephone contact 3 times, and left a callback number when possible. When telephone contact was successful, the care manager attempted to identify and resolve any barriers to obtaining the service by providing education and, when appropriate, by offering needed referrals, facilitating the scheduling of necessary appointments, or referring the patient back to the practice clinicians if questions arose.

	All measurements were obtained using automated searches of EHR-derived data. The primary outcome for each patient was the time from refusal to receipt of any refused preventive service (colorectal cancer screening, breast cancer screening, cervical cancer screening, osteoporosis screening, or pneumococcal vaccination). This included (1) the result of a test or vaccination performed at this center or (2) clinician documentation in a standard field of the EHR that the test had been performed elsewhere. For patients with recorded refusals of more than 1 preventive service, the one with the shortest time from refusal to completion was used in the primary analysis. We also examined completion of the individual preventive service separately. We recorded the time spent by the care manager performing outreach.

	We compared time to receipt of the refused sevice for the outreach cohort and for the nonintervention cohort using curves obtained using the Kaplan-Meier method. We used Cox regression models to calculate proportional hazards adjusted for patient characteristics (age, sex, race/ethnicity, and insurance type as categorized in 

 when appropriate). Analyses used commercially available statistical software (STATA version 10; StataCorp LP, College Station, TX).

	The study was funded by the Agency for Healthcare Research and Quality. The funder had no role in the design, conduct, or analysis of the study or in the decision to submit the manuscript for publication.

	The characteristics of patients who refused preventive services in both cohorts are listed in Table 1. In the outreach cohort, 407 patients had 1 or more refusals documented for 520 preventive services. This represented approximately 3.4% of the practice population that was eligible for 1 or more preventive services. In the nonintervention cohort, 378 patients had 1 or more refusals documented for 510 preventive services. Compared with the nonintervention cohort, the outreach cohort was slightly older, was more likely to have Medicare insurance, and more frequently had refused colorectal cancer screening or pneumococcal vaccination.

	We did not perform outreach for 23 of 407 patients (5.7%) because they were not included in the weekly patient outreach lists. We deemed outreach to be inappropriate for 71 of 384 patients (18.5%) after EHR review. Primary care physicians requested by e-mail that no outreach be performed for 41 of 384 patients (10.7%). We mailed outreach material and attempted telephone contact for 247 of 384 patients (64.3%) in the outreach cohort. Telephone contact was ultimately successful for 46 patients. The remainder did not answer telephone calls and did not respond to phone messages requesting that they call back.

Completion of Refused Preventive Services

	Few patients received any preventive service within 6 months of when the refusal was recorded in the EHR. There was no difference between cohorts in the receipt of any refused preventive service (6.1%; 95% confidence interval [CI], 4.0%-8.9% of the outreach cohort vs 4.8%; 95% CI, 2.8%-7.4% of the nonintervention cohort (adjusted hazard ratio, 1.3; 95% CI, 0.7-2.5) (

). Table 2 gives the numbers and percentages of patients in the 2 cohorts who refused services and the rates of receiving each service within 6 months. The outreach cohort and the nonintervention cohort differed in receipt of services. However, the number of patients who received any individual service was generally small, and estimates of the effect of outreach for individual services are imprecise.

	Receipt of any refused service within 6 months occurred in 8.1% of patients for whom no outreach was performed, in 2.9% of patients who were mailed outreach materials but could not be contacted by phone, and in 13.0% of patients for whom phone contact was ultimately successful. These differences were significant (

	Conducting outreach for 407 patients with documented refusal of 1 or more preventive services required 214 hours of care manager time. This included EHR reviews, e-mails to physicians, preparation of mailings, attempted and completed phone calls, and appointment coordination.

	Patients have the right to decline unwanted preventive care. However, if their refusals are based on inadequate information, patients may be making choices that differ from what they would decide if they were more fully informed. We provided an educational outreach intervention to patients who refused a preventive service recommended by their physician to ensure that patients received the key information that justified the use of the preventive service. We also gave an opportunity for patients to address perceived barriers that may have led them to refuse. We found no significant increase in the eventual receipt of preventive services using this approach over a 6-month time frame. Furthermore, the resources required to perform outreach were substantial.

	Because few patients received previously refused services in this study, our estimates of the effects of outreach are imprecise, and our 95% CI for the primary outcome does not exclude up to a 2.5-fold increase in the use of preventive services attributable to outreach. Larger studies could provide more definitive results, or a randomized controlled trial testing these methods could be performed. However, within the confines of our study size, we believe that several relevant observations can be made about the difficulty in promoting preventive services use among individuals who have refused it. Under the best of circumstances using the data herein, a 2.5-fold increase in preventive services use would amount to only a 7.2% absolute increase in the use of refused services because the underlying rate of service use among those who had refused was so low. It is more likely that, if there is an effect of educational outreach on patients who have refused services at all, it would not lead to more than a few percentage points of absolute increase in the receipt of services among this group.

 indicate that less intensive patient-directed interventions, such as mailed invitations or reminders, can increase the use of preventive services among general populations who are due for them. Our findings suggest that these approaches, while they may increase appropriate preventive services use for the population as a whole, may not have an important influence on known refusers. Furthermore, patientswho refused services in this setting were difficult to contact by phone, and most did not return care manager calls. Healthcare providers who perform educational outreach among populations to promote the use of preventive services could consider whether recorded patient refusals captured within an EHR should be used as a way to identify patients who are not likely to respond to a simple intervention (ie, mailed reminder). Therefore, resources could be conserved by not performing outreach directed at this group whose lack of acceptance of the preventive service may be difficult to change.

	Cervical cancer screening may be an area in which further study of this outreach approach is warranted. Our results indicated that the overall rate of receiving cervical cancer screening within 6 months of refusing was higher than for other services, and the measured effect of outreach on cervical cancer screening was the largest among 5 services examined. Given our limited study size, we cannot say whether these findings are simply due to chance.

	Our findings have implications for performance measurement programs used for holding providers accountable. If few patients change their mind once they refuse a preventive service even after receiving extra educational interventions, there may be little that healthcare providers can do to influence measured quality for these subpopulations. Recording patient preferences in the case of refusals and incorporating these into performance measures may be necessary to prevent accountability programs from inadvertently providing incentives to providers to avoid patients who refuse recommended treatments.

 Our study adds to prior studies demonstrating that reporting exceptions to performance measures (eg, medical contraindications, adverse effects, or patient preferences) may lead to changes in the conclusions of quality reports,

 and that exceptions to performance measures in many cases seem justified.

	In addition to the size limitation aforementioned, our study has some other limitations. It was not a randomized controlled trial. While the 2 cohorts were similar and although we adjusted for some potential confounders, the possibility of residual confounding remains. We examined the effect of only a 1-time outreach intervention performed by a nonclinician. We did not address all potential reasons patients may have for refusing; the intervention was designed to provide patient education and to refute common misconceptions, but there may have been other cultural beliefs held by patients that were not addressed. We do not know if repeated outreach or outreach conducted in a different fashion would have produced different results. We did not formally assess patients’ reactions to the outreach, so we do not know if patients were bothered by being contacted about healthcare services that they had already discussed and had told a physician they did not want. Because our study was performed at a single large site, we do not know how well these findings generalize to other practice settings.

Educational outreach did not seem to be a promising strategy to promote preventive services use among patients who refused services recommended by their physician.

Stephen D. Persell, MD, MPH

Articles

Implementation of EHR-Based Strategies to Improve Outpatient CAD Care

Effects of Standardized Outreach for Patients Refusing Preventive Services: A Quasiexperimental Quality Improvement Study