Susan M. Shetterly, MS

To examine the impact of enrolling in a healthcare plan through the Affordable Care Act (ACA) healthcare exchanges on self-reported access to care. 

Cohort study using self-reported data of patients newly enrolled in Kaiser Permanente California and Kaiser Permanente Colorado through the ACA healthcare exchanges for coverage beginning January 1, 2014. 

 Baseline and follow-up surveys conducted via mail and telephone, with response rates of 45% and 51%, respectively. 

 We found significant increases in the percentage of people who reported having a personal healthcare provider (59% vs 73%; 

 <.01) and significant decreases in those who reported delaying needed medical care due to costs (37% vs 25%; 

 <.01) before and after ACA enrollment. There was also a significant increase in the percentage of patients who reported receiving a flu shot during the prior year (41% vs 52%; 

 <.01). Among the people who reported having less than 4 months of healthcare coverage in 2013, these improvements were even more pronounced. This group also showed significant increases in the percentages who felt they had a place to go when they needed medical care (43% vs 56%; 

 <.01) and who reported they received advice to quit smoking or using tobacco (46% vs 72%; 

These findings are an important addition to the evidence base that the ACA is improving the healthcare experience and reducing barriers due to costs for individuals obtaining insurance coverage through the healthcare exchanges.

Among patients newly enrolling in Kaiser Permanente California and Colorado through the Affordable Care Act (ACA) healthcare exchanges, we found: 

		Significant increases in the percentage of people who reported having a personal healthcare provider. 

		Significant decreases in the percentage of people who reported delaying needed medical care due to costs.

		Significant increases in the percentage who felt they had a place to go when they needed medical care and who received advice to quit smoking.

These findings are an important addition to the evidence base that the ACA may be improving access to care for people obtaining insurance coverage through the healthcare exchanges.

A major goal of the Affordable Care Act (ACA) is to increase access to healthcare, particularly for those who are uninsured, underinsured, or without stable healthcare insurance.

 The ACA has reduced the size of the uninsured population, and in 2016, close to 13 million Americans enrolled in a health plan through the federal or state-operated healthcare exchanges.

 However, there has been significant debate as to whether the ACA has achieved its goals of improving access and decreasing costs and whether it should continue in its current form.

 Most empirical studies since implementation of the ACA’s healthcare exchanges in 2014 have focused on the number of uninsured Americans within the United States and by state

 or examined care trends at the population level.

 The impact of insurance obtained through healthcare exchanges on access and costs of care at the individual level is largely unknown. The purpose of our study was to measure changes in self-reported healthcare access and cost-related barriers to healthcare among patients obtaining insurance through the ACA’s healthcare exchanges.

We surveyed 2145 adults aged 18 to 64 years who obtained insurance coverage through Kaiser Permanente (KP) by enrolling via the California or Colorado state healthcare exchanges starting on January 1, 2014. The survey was restricted to individuals who had not had KP healthcare coverage in the 5 years prior to initial exchange enrollment. Survey questions asked about healthcare experiences in the prior year and individuals' perceptions of whether they had a usual source of care and a regular healthcare provider or had delayed healthcare due to cost. Questions also assessed patient self-reported preventive behaviors and health behavior counseling. New KP members eligible for the survey were initially contacted by mail and asked to return a paper version of the survey; nonrespondents were called and offered the survey by phone. Responders to the initial baseline survey were re-contacted in 1 year and asked the same set of questions in order to assess if access, preventive behaviors, or cost barriers had changed since healthcare exchange enrollment. The response rate for the baseline survey was 45%, and the response rate for the follow-up survey was 51%. We compared differences in individual baseline survey responses regarding experiences in the year prior to ACA enrollment (2013) with follow-up survey responses regarding experiences in the year post-ACA enrollment (2014) at the individual level using χ

 tests for significance. Analyses were performed using SAS version 9.2 (SAS Institute, Cary, North Carolina). This study was approved by the KP Northern California and KP Colorado institutional review boards. 

A total of 528 patients completed both the baseline and follow-up surveys; these responders were 55% female and 71% non-Hispanic white, with 62% reporting a household income of less than $50,000 per year (data not shown). We found significant increases in the percentage of people who reported having a personal healthcare provider (59% vs 73%; 

 <.01) and significant decreases in the percentage of people who reported delaying needed medical care due to costs (37% vs 25%; 

 <.01) before and after ACA enrollment. There was a significant increase in the percentage of patients who reported receiving a flu shot as well (41% vs 52%; 

). Among the individuals who reported having fewer than 4 months of healthcare coverage in 2013, these improvements were even more pronounced (

). In addition, healthcare exchange enrollment was associated with significant increases in the percentage who felt they had a place to go when they needed medical care (43% vs 56%; 

 <.01) and who reported receiving advice to quit smoking or using tobacco (46% vs 72%; 

We found that adults obtaining insurance through the healthcare exchanges reported increased access to care, increased use of preventive care, and fewer barriers to medical care due to cost. These positive changes were even greater among patients with fewer than 4 months of healthcare coverage during the year prior to enrollment. One prior study found improved trends in access and affordability after the ACA’s enactment by examining population-level trends

; our analysis adds significant strength to the argument that the ACA has had a positive impact on care access, receipt of preventive services, and costs by examining changes in patient experiences at the individual level. Although our study was conducted among ACA enrollees in systems whose cost-sharing structures might not represent all exchange plans, our finding that delays in receiving needed medical care were reduced after enrolling through the ACA exchanges is particularly important in light of the fact that many exchange insurance products offered are high-deductible health plans

 that involve significant cost sharing for patients until the deductible is met. 

A major goal of the ACA is to help American healthcare achieve the “Triple Aim” of simultaneously optimizing healthcare access and quality while reducing costs.

 These findings are an important addition to the evidence base that the ACA may be improving the healthcare experience, reducing barriers due to costs, and contributing to a greater sense of engagement with the healthcare system

 in people obtaining insurance coverage through the healthcare exchanges.

 Division of Research, Kaiser Permanente Northern California (JAS, DW, LM), Oakland, CA; Institute for Health Research, Kaiser Permanente Colorado (JCB, SMS), Denver, CO; Kaiser Permanente (DS), Santa Clara, CA; School of Public Health, University of California at Berkeley (CVC), Berkeley, CA

 This study was funded by the Kaiser Permanente Center for Effectiveness and Safety Research, Contract no. KR021125. Dr Schmittdiel also received support from the Health Delivery Systems Center for Diabetes Translational Research (CDTR) [NIDDK grant 1P30-DK092924].

 Drs Schmittdiel and Sam, and Ms Barrow and Ms Shetterly are employed by Kaiser Permanente. Ms Barrow and Ms Shetterly received grant funding for this project from the Kaiser Permanente national office. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

 Concept and design (JAS, JCB, DS); acquisition of data (JAS, JCB, CVC, SMS); analysis and interpretation of data (JAS, LM, DS, SMS); drafting of the manuscript (JAS, JCB, DW, LM); critical revision of the manuscript for important intellectual content (JCB, DS, CVC, SMS); statistical analysis (JAS, LM); provision of patients or study materials (DW); obtaining funding (JAS, JCB); administrative, technical, or logistic support (JCB, DW, CVC); and supervision (JAS). 

 Julie A. Schmittdiel, PhD, Kaiser Permanente Northern California, 2000 Broadway, Oakland, CA 94612. E-mail: Julie.A.Schmittdiel@kp.org. 

1. Blumenthal D, Abrams M, Nuzum R. The Affordable Care Act at 5 years. 

 2015;372(25):2451-2458. doi: 10.1056/NEJMhpr1503614.

2. Jost T. The 2016 open enrollment tally: 12.7 million selected plans, 1 million more than 2015.  

 Blog website. http://healthaffairs.org/blog/2016/02/04/the-2016-open-enrollment-tally-12-7-million-selected-plans-1-million-more-than-2015/. Published February 4, 2016. Accessed February 11, 2016.

3. Berwick DM, Feeley D, Loehrer S. Change from the inside out: health care leaders taking the helm. 

2015;313(17):1707-1708. doi: 10.1001/jama.2015.2830.

4. Carman KG, Eibner C, Paddock SM. Trends in health insurance enrollment, 2013-15. 

2015;34(6):1044-1048. doi: 10.1377/hlthaff.2015.0266.

5. Sommers BD, Buchmueller T, Decker SL, Carey C, Kronick R. The Affordable Care Act has led to significant gains in health insurance and access to care for young adults. 

2013;32(1):165-174. doi: 10.1377/hlthaff.2012.0552.

6. Sommers BD, Gunja MZ, Finegold K, Musco T. Changes in self-reported insurance coverage, access to care, and health under the Affordable Care Act. 

2015;314(4):366-374. doi: 10.1001/jama.2015.8421.

7. Wharam JF, Ross-Degnan D, Rosenthal MB. The ACA and high-deductible insurance—strategies for sharpening a blunt instrument. 

2013;369(16):1481-1484. doi: 10.1056/NEJMp1309490.

8. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. 

2008;27(3):759-769. doi: 10.1377/hlthaff.27.3.759.

9. Fisher ES, McClellan MB, Bertko J, et al. Fostering accountable health care: moving forward in Medicare. 

2009;28(2):w219-w231. doi: 10.1377/hlthaff.28.2.w219.

Articles

Affordable Care Act exchange enrollees in California and Colorado reported significant improvements in access to care and fewer barriers to receiving care due to costs.

Improvements in Access and Care Through the Affordable Care Act

To evaluate a population outreach program to promote screening for colorectal cancer (CRC) among average-risk insured men and women.

In 2008, 58,440 Kaiser Permanente Colorado members unscreened for CRC received an interactive voice response (IVR) call followed by mailed fecal immunochemical test (FIT), or colonoscopy if requested. We used a quasi-experimental design with staged implementation, in which a random subset of eligible members was selected each week to receive the intervention. This design allowed the entire group to ultimately receive the intervention.

Survival models summarized timespecific comparisons of screening behaviors for members who received immediate outreach compared with those who had not yet received it.

A total of 26,003 (45%) of the unscreened population completed screening, predominately due to the mailed kits. The unadjusted hazard ratio (HR) for the outreach effect on screening completion was 4.08 (95% confi dence interval: 3.93-4.25) and adjusted HR was 3.75 (3.60-3.91). Lower levels of screening were seen in African Americans (HR 0.83; 0.77-0.90) and Hispanics (HR 0.84; 0.80-0.88) compared with whites, and in smokers (HR 0.77; 0.74-0.80) compared with nonsmokers. The outreach had greater impact among those without a primary care (HR 4.5 vs 3.0, 

 <.0001) or specialty care (HR 5.2 vs 3.5,

 <.0001) visit compared with those with 1 or more visits.

 The rate of colorectal cancer screening in members after mailed FIT with IVR was almost 4 times higher than usual care, particularly in those without an office visit. Targeted approaches are needed for groups at risk for not screening.

We used a quasi-experimental design to evaluate a large outreach program in an insured population to promote colorectal cancer screening by stool testing or colonoscopy.

		 Unscreened members received an educational automated call and mailed immunochemical stool test.

		 Forty-five percent of unscreened members completed screening within the study period.

		 The outreach had even greater impact among members without a primary care visit (HR 4.5 vs 3.0, P <.0001) or specialty care visit (HR 5.2 vs 3.5, 

<.0001) compared with those with 1 or more visits.

		 Targeted approaches are additionally needed for groups at risk for not screening.

Mortality from colorectal cancer (CRC) is declining, yet it remains the second-leading cause of cancer death in the United States and the leading cause among nonsmokers. Screening of average-risk men and women aged 50 to 75 years is recommended by the US Preventive Services Task Force (USPSTF) as well as specialty organizations because early detection of high-risk adenomas and cancers is associated with decreased CRC incidence and mortality.

 Screening with fecal occult blood testing and sigmoidoscopy have been shown in clinical trials to decrease CRC mortality. 

 Evidence of effectiveness for screening colonoscopy remains indirect,

 though 1 recent case-control study did show a signifi cant benefit.

Screening rates for CRC lag behind rates for breast and cervical cancer and vary by socioeconomic status, insured status, and ethnicity.

The age-adjusted prevalence for screening (fecal occult blood test [FOBT] within a year or lower endoscopy within 10 years) in 2010 was 65.4% for adults aged 50 to 75 years, up from 52.3% in 2002.

The ability to screen large numbers of individuals for CRC in a cost-effective manner is critical as states and large health plans assume more responsibility for aging populations. The fecal immunochemical test (FIT) offers increased sensitivity for cancer and specifi city compared with standard guaiac FOBTs. Thus FIT is a non-invasive low-cost option which may achieve higher levels of population acceptance than primary colonoscopy. 

 The ability to process large numbers of samples by machine facilitates mass screening. This paper describes our quasi-experimental evaluation of a large outreach program using an interactive voice response (IVR) call followed by mailed FIT, or colonoscopy if requested, to increase screening among average-risk men and women in a nonprofit integrated care delivery system.

Kaiser Permanente Colorado (KPCO) is a not-for-profit integrated care delivery system with over 530,000 members. KPCO uses an electronic medical record (EMR) and multiple population registries for prevention and chronic disease management programs. The CRC registry includes family history, diagnoses, procedures, and pathology dating back to 1994. For this study, the registry was used to determine eligibility in April 2008 for CRC screening in members who had continuous enrollment until April 2009, the end of the study period. The study protocol was approved by the Institutional Review Board and Research Review Committees of KPCO. The need for individual signed consent was waived.

In fall 2007, KPCO replaced the 3 sample guaiac FOBTs with the 1 sample FIT from Polymedco (Cortland Manor, New York). From April through mid-September 2008, a systematic outreach program targeted all unscreened average-risk men and women at a rate of 3000 IVR calls a week (

). The call lasted approximately 5 minutes and included options for education about screening modalities. Members could clarify risk status and, if eligible, request a mailed FIT kit as well as rank their perception of barriers to screening. The registry was updated with new information regarding high-risk status, and the member’s provider was notifi ed for colonoscopy referral if requested. Average-risk members who completed the call and requested stool testing were mailed a kit within 2 weeks. Kits included an invitational letter with information about stool testing, an FIT kit with instructions, and a prepaid return envelope. A section in Spanish gave a phone number to call for assistance in Spanish. Members who did not complete the IVR call were mailed the FIT kit 30 days later. If the kit was not returned, a reminder letter was sent at 4 weeks. Members with negative FIT results were notified by mail; positive results were sent electronically through the medical record to the primary care provider for referral. Members with positive studies who did not complete diagnostic evaluation received a reminder letter at 8 weeks followed by a certifi ed letter at 16 weeks. The program sent yearly FIT tests on the anniversary date of initial testing for members who chose to screen by FIT and remained eligible.

Both the IVR and outreach letter also offered colonoscopy as a screening option. Members were informed that not all cancers are found by stool testing and that colonoscopy would be required if their FIT test was positive. Members who chose colonoscopy for screening were asked to contact their provider for a referral and the outreach program notifi ed the provider through the EMR.

Efforts to increase screening began before 2008 and included provider and staff education, new promotional materials for members, panel screening reports, and quality incentive programs. The CRC registry prompted in-reach alerts printed on individual check-in sheets in primary care visits for all ageand risk-eligible members, stating “colorectal screening may be due” and providing a phone number for members to report prior screening or risk status. This in-reach alert continued until there was documentation of screening refusal, a completed FIT, or colonoscopy pathology.

A summary of study exclusions and the flow of the outreach for FIT testing is shown in Figure 1. Initial exclusions included high-risk status, removal from the registry by personal physician due to comorbidities, or evidence of completed screening. High-risk status included personal history of CRC or polyps, family history (first-degree relative) of CRC, infl ammatory bowel disease, and genetic syndromes. Average-risk members were assumed to be unscreened if they did not have a completed FOBT or FIT within 12 months or a colonoscopy within 10 years. Members who had had barium enema or fl exible sigmoidoscopy within 5 years were eligible for FIT outreach, as the combination of FIT with either was judged to improve screening effectiveness.

 The CRC registry tracked members according to their risk status and most recent screening modality. All average-risk members who were unscreened in April 2008 and were continuously enrolled until April 2009 were used for the analysis reported here (N = 58,440).

We used a quasi-experimental design with staged implementation, in which a random subset of eligible members was selected each week to receive the intervention. This design allowed the entire group to ultimately receive the intervention, while not exceeding the capacity of providers and systems during any single time period. The timing of outreach was randomized by medical record number. Survival models summarized time-specifi c comparisons of screening behaviors for members, comparing exposed and unexposed person-time, with each individual contributing a variable amount to the immediate to the delayed intervention control groups. This analytic

strategy isolated the effect of the outreach from the effects of usual care received by all members during the study period. While all members received the outreach between April and September 2008, the follow-up period extended an additional 6 months until April 2009. All members eligible at the start of outreach were included and analyzed by intention to treat.

The analysis was done in 2010 and used SAS 9.13 (SAS Institute, Cary, North Carolina). The primary outcome was a 2-category variable: screened by FIT or colonoscopy versus remaining unscreened April 2008 through April 2009. Analysis variables included age, gender, race/ethnicity, smoking status, body mass index (BMI), enrollment in a chronic disease registry, and visit history within the past year. Cox proportional hazards models estimated the time to screening completion using the time of the IVR call, which initiated the outreach, as a time-varying covariate.

 In addition to main effects, interaction terms examined whether the outreach effectiveness varied across race/ethnic subgroups and by evidence of having had a visit in primary care, obstetrics/gynecology (Ob/Gyn), or specialty care during the past year.

We also conducted secondary univariate analyses to compare characteristics of those choosing FIT versus colonoscopy during the study period. To evaluate sustainability, a descriptive analysis identifi ed members who completed FIT screening after the initial outreach, were still enrolled and eligible on their year 2 anniversary date, and completed a second FIT.

The characteristics of the entire study population are shown in 

; this table does not compare intervention and control groups because, by design, all members ultimately received the intervention after a variable time delay. The study group of 58,440 included 8625 members who did not receive IVR/FIT mailing due to previously  unknown screening or administrative data changes. A total of 26,003 (45%) completed screening: 3497 screened with FIT by request through the IVR call; 14,952 through mailonly FIT; 5255 screened by colonoscopy without prior FIT; and 1299 screened by FIT via usual care. The mean age was 58.8 years and 53% were female. Members aged 65 to 74 years were more likely to complete screening than members aged 50 to 64 years (

 <.001). There were signifi cantly lower rates of screening (

 <.001) for African Americans, Hispanics, active smokers, members with increased BMI, and those missing demographic data. Members classifi ed as having a chronic condition in KPCO disease registries had higher rates of screening than members not in disease registries. Screening completion was higher with a history of at least 1 visit in primary care, Ob/Gyn, or specialty departments. While 83% of members had at least 1 visit during the first year, 43% of members without any primary care visit during the year completed screening.

Cox proportional hazards models were used to evaluate the impact of the outreach program on screening with either FIT or colonoscopy (

). The unadjusted hazard ratio indicated a 4-fold (HR 4.09; 95% CI: 3.93-4.25) increase in screening compared with the delayed intervention control group; the effect was reduced after adjustment for selected patient factors (HR 3.75; 3.60-3.91). Having missing demographic data in any category, refl ecting less engagement with care delivery, was associated with decreased response to outreach. Signifi cant interactions (

 <.0001) were found between the outreach effect and visit history (

). The impact of the outreach was greater for members with no visit in primary care (HR 4.47; 4.24- 4.72) compared with members with 1 or more visits (HR 3.0; 2.86-3.20). For members with specialty care visits, the impact of the outreach was even higher, with an HR of 5.16 (4.69-5.68) with no visit and 3.54 (3.39-3.70) with 1 or more visits. There was no signifi cant interaction by race/ethnicity (

>.4 for African Americans and Hispanics) on the effect of the outreach.

The IVR calls were completed by 17% of outreached members. The 182 members who self-identifi ed in the IVR as already screened or high risk were not mailed an FIT kit. Nearly 80% of those who completed CRC screening through the outreach chose FIT and there were no significant differences in screening modality choice by gender or ethnicity. Although FIT was the predominant choice of both younger and older members, younger members chose colonoscopy more often than older members (23% compared with 16%). Members screened by colonoscopy were also more likely to have done so if they had had any office visit. By the end of the study period, 79% of those with positive FIT results had undergone colonoscopy; the remainder continued to receive active outreach up to and including certified letters. In the second year of the program, 9435 (47.8%) of the original 19,748 who chose FIT in year 1 and were still eligible completed it again.

Automated phone and mail population outreach resulted in an almost 4-fold increase in the rate of screening for colorectal cancer even without an office visit.

Automated Phone and Mail Population Outreach to Promote Colorectal Cancer Screening

To assess the effect of a targeted age-specific computerized alert to reduce D-dimer testing in elderly patients.

 A single-crossover cluster randomized trial of computerized alerts during physician order entry involving 8 ambulatory care clinics in a group-model integrated care delivery system.

 The rate of completed D-dimer tests per 1000 patient visits, ratio of completed venous ultrasonography to completed D-dimer tests, and rate of completed venous ultrasonography per 1000 patient visits.

 The rate of completed D-dimer tests per 1000 visits among patients 65 years and older in intervention clinics decreased from 5.02 to 1.52 (95% confidence interval [CI], -4.20 to -2.80;

 <.001), which persisted throughout the study period. The rate of completed D-dimer tests per 1000 visits among patients 65 years and older in control clinics decreased from 3.14 to 2.11 (95% CI, -1.66 to -0.04; 

 <.001 for interaction). After activation of the alert in control clinics, the rate of completed D-dimer tests per 1000 visits among patients 65 years and older decreased from 2.11 to 0.81 (95% Cl, -1.79 to -0.80; 

 <.001). After activation of the alert in each clinic group, the ratios of completed venous ultrasonography to completed D-dimer tests increased from 1.17 to 4.05 (95% CI, 2.52-3.22) and from 2.25 to 7.29 (95% CI, 3.74-6.35) in intervention clinics and control clinics, respectively (

 An electronic age-specific alert targeted to a specific condition reduced D-dimer testing in this elderly population of outpatients and demonstrated a persistent effect.

	Physicians often find it difficult to remember to follow evidence-based clinical practice guidelines. Many researchers have suggested that computer-generated alerts within electronic medical records may serve as reminders to improve adherence to best practices. However, too many alerts produce alert fatigue and may lead physicians to ignore them. Our study shows that an alert targeted to a specific order and for specific patients can alter a physician's ordering behavior and promote improved adherence to a clinical practice guideline.

		A targeted computerized alert reduced the use of D-dimer testing in elderly patients.

		An alert targeted to a specific condition demonstrated a persistent effect.

		Computerized alerts should contain alternative diagnostic or treatment strategies to direct clinicians toward more appropriate diagnostic strategies rather than just provide "negative guidance."

	The risk of developing a blood clot in the venous circulation increases exponentially with age from approximately 30 cases per 100,000 persons aged 25 to 35 years to 300 to 500 cases per 100,000 persons aged 70 to 79 years.

 Rates of pulmonary embolism (PE) increase from 120 per 100,000 persons aged 65 to 69 years to approximately 700 per 100,000 persons 85 years and older; about half of all deaths after venous thromboembolism (VTE) are attributed to PE.

 Venous thromboembolism most commonly includes deep vein thrombosis (DVT) or PE. Symptoms common to PE and DVT appear in various clinical conditions. Clinical diagnosis of suspected acute VTE is difficult. Therefore, objective testing combined with a clinical risk algorithm is recommended.

	Since the late 1980s, high-resolution venous ultrasonography has become widely used to diagnose or exclude DVT in most community practices throughout the United States.

 In the early 1980s, a quantitative D-dimer test based on enzyme-linked immunosorbent assay (ELISA) was introduced as a means to exclude acute VTE. Investigators have studied the role of D-dimer in the diagnosis of DVT and PE.

using a sensitive automated ELISA D-dimer assay have shown that D-dimer levels below an established cutoff of 500 ng/mL combined with a clinical risk algorithm enable exclusion of acute venous thrombosis in up to 30% of outpatients (to convert D-dimer level to nanomoles per liter, multiply by 5.476). However, the negative predictive value of the test worsens as patient age increases.

 Although the sensitivity and the negative predictive value of the test are about 98%, the specificity is less than 30%, and the overall accuracy of the test is only 35% for patients 65 years and older.

 Previously, our group demonstrated that more of these tests are ordered in clinical practices where D-dimer testing is readily available.

 In real-world experience, it was found that more than 73% of all D-dimer tests among patients 65 years and older had positive results, and almost 65% of these were false positives that required additional radiologic evaluation. Only about 27% of all D-dimer tests performed among the elderly had negative results; therefore, only these patients avoided additional radiologic procedures. In comparison, among patients younger than 65 years, the test had a negative predictive value exceeding 99%, a specificity exceeding 70%, and an overall accuracy exceeding 70%.

	In 2007, the American Academy of Family Physicians and the American College of Physicians

 jointly published a guideline onthe diagnosis of lower extremity DVT and PE, which recommended the use of clinical prediction rules to establish pretest probability of disease and advised using D-dimer testing to exclude VTE in younger patients without associated comorbidity or history of VTE. However, the guideline notes that D-dimer testing may be insufficient to rule out VTE in older patients.

 Other researchers also present evidence supporting that older patients should not receive D-dimer testing and recommend that an alternative strategy based on noninvasive examinations such as venous ultrasonography should be the first choice for evaluation of possible DVT.

	In 2001, Kaiser Permanente of Colorado adopted the quantitative D-dimer test for use in clinical practice. Since then, the number of D-dimer tests ordered for patients 65 years and older has increased 5-fold, accounting for about one-third of all D-dimer tests ordered. The convenience of ordering a laboratory test and of potentially avoiding a trip by the patient to a diagnostic radiology center may be partially fueling the increased use of the D-dimer test. However, because of the quantity of tests being ordered in patients 65 years and older and owing to poor test performance in this age group, we have also seen a significant increase in the number of positive D-dimer test results, which required additional testing using radiologic evaluation. Most positive D-dimer test results in the elderly (65%) were determined to be false positives on further evaluation of patients’ venous ultrasonography results.

Therefore, we wondered whether a computerized alert within the physician order entry module of the electronic medical record (EMR) that we use (Kaiser Permanente HealthConnect [KPHC]; Epic Systems, Verona, Wisconsin) could reduce the number of D-dimer tests ordered for elderly patients. We hypothesized that computergenerated reminders given to primary care providers at thetime of ordering a D-dimer test for patients 65 years and older for suspected VTE would decrease the rate of D-dimer test ordering among providers receiving the alert compared with providers not receiving the alert. This article reports our results of a randomized controlled study using this alert within a group-model health maintenance organization using an EMR for documentation and order entry of all outpatient clinical care.

	Between September 2005 and October 2007, we performed an internally funded cluster randomized trial involving 8 ambulatory care clinics in a group-model integrated care delivery system in Denver, Colorado. The institutional review board of Kaiser Permanente of Colorado approved the study before the start. This research did not require contact with study subjects (physicians or patients). Data used were retrospective and were extracted in a deidentified format before analysis began. This trial began before July 1, 2008, and did not require trial registration to meet guidelines of the International Committee of Medical Journal Editors.

	We identified 8 primary care clinics, each with at least 3000 patients 65 years and older. We performed a simple cluster randomization to assign half of the clinics to a control group and the other half to an intervention group. A cluster randomized design was selected to decrease the risk of interprovider contamination among providers within the same facility. Primary care providers within the designated clinics were unaware that a research study was occurring in regard to D-dimer test ordering related to the alerts.

	All primary care providers received training and information about the use of clinical risk algorithms for patients with suspected venous thrombosis and about the performance characteristics of the D-dimer test among various age groups.

 This training took place during one of the monthly continuing medical education meetings for primary care providers 4 months before the intervention. Primary care providers in intervention clinics also saw the following pop-up alert within the physician order entry system at the time of placing a computerized D-dimer test order for patients 65 years and older, advising against ordering the D-dimer test: “D-dimer testing is not recommended for patients 65 years

	of age and over because the accuracy is only 35% in this age group. Instead, order diagnostic imaging as appropriate.” Providers in control clinics did not receive this alert during the initial intervention. After 20 months, the alert was activated in control clinics; it remained active in intervention clinics.The specific metrics for each step of the project are shown in 

	Data on D-dimer tests and venous ultrasonography among patients having office visits in intervention clinics and control clinics were collected before and after activation of the alert. Data included clinic site, date of D-dimer order and result, and date of radiology test (

 code 93971 [ultrasonography of extremities]) and result after a D-dimer test. Except for patient age, other patient characteristics were unavailable in the data sets used for analyses.

	The primary outcome measure was the number of D-dimer tests ordered and completed per 1000 patient visits (including all office visits for all patients in each age group for each clinic). This rate excluded D-dimer tests that occurred within 14 days of each other for the same patient. The secondary outcome measure, the effect on ultrasonography ordering for evaluation of DVT, was assessed by determining the ratio of completed venous ultrasonography to completed D-dimer tests and the rate of completed venous ultrasonography per 1000 patient visits for each age group.

	Outcome variables of interest were counts (eg, D-dimer tests estimated relative to a number of visits to account for differing ordering opportunities). Descriptive rates and Fisher exact test confidence intervals (CIs) were calculated for intervention groups and control groups for each period. Count data are often modeled using Poisson distribution regression analysis, but negative binomial models were a better fit because of the overdispersion evident among these data.

 Models were run using commercially available software (GLIMMIX program in SAS; SAS Institute Inc, Cary, North Carolina), with the numerator of the rate as the outcome and a logged value for the denominator used as an offset term. Analysts were unaware of the clinic assignments during the data collection steps, but were aware of the clinic status during the statistical analysis phase. Each clinic had 2 periods of interest (preintervention and postintervention), and the test of intervention effectiveness was evaluated using the interaction term of the intervention variable with time. This test evaluates if the change in the outcome variable over time was significantly different for intervention clinic versus control clinic groups. Estimates of clinic-level intraclass correlations were low (<0.01), and the addition of a random-effects term for facility clusters in the negative binomial models produced comparable results. The minimal influence on model results suggests limited clustering effects within the specific clinics.

	The mean age of patients 65 years and older receiving D-dimer testing was 75.6 years in intervention clinics and 75.7 years in control clinics. The mean age of patients younger than 65 years receiving D-dimer testing was 48.7 years in intervention clinics and 49.1 years in control clinics. Among patients 65 years and older, the baseline rates during 8 months before the study for completed D-dimer tests per 1000 clinic visits was 5.02 in intervention clinics and 3.14 in control clinics (rate difference, 1.88; 95% CI, 1.03-2.74; 

<.001). After the alert was activated in intervention clinics, the rate of completed D-dimer tests per 1000 clinic visits for patients 65 years and older (September 2005 through April 2007) dropped significantly in all clinics, but the rate dropped significantly more in intervention clinics than in control clinics (from 5.02 to 1.52; 95% CI, −4.20 to −2.80; 

<.001 in intervention clinics and from 3.14 to 2.11; 95% CI, −1.66 to −0.40; 

). The rate differences between intervention clinics and control clinics were 1.52 and 2.11, respectively (rate difference, −0.59; 95% CI, −0.98 to −0.19; 

	Activation of the alert in control clinics was followed by a significant decrease in the rate of completed D-dimer tests per 1000 clinic visits for patients 65 years and older in these clinics. The rate decreased from 2.11 to 0.81 (95% CI, −1.79 to −0.80; 

< .001). During the period when the alert was active in all 8 clinics (May through September 2007), the rate of completed D-dimer tests per 1000 clinic visits for patients 65 years and older was 1.89 in intervention clinics and 0.81 in control clinics (rate difference, 1.08; 95% CI, 0.42-1.74;

 = .002). After the initial rate change in intervention clinics, the new lower rate was maintained, indicating persistent effectiveness of the alert over time (Table 1 and Figure 2).

	The baseline rates of completed D-dimer tests per 1000 clinic visits for patients younger than 65 years were not significantly different between intervention clinics and control clinics. The rate of completed D-dimer tests per 1000 clinic visits in control clinics for patients younger than 65 years did not significantly change over the course of the study; the rate for these patients in intervention clinics did not initially change after the alert was established but showed a slight increase by the end of the study period (Table 1). As the alert was not active for patients in this age group, these rates indicate that physician decision making for these patients was unaffected.

	Because the alert advised against ordering a D-dimer test and recommended using a radiologic test (ie, venous ultrasonography) to evaluate patients 65 years and older for suspected DVT, we expected to see an increase in the rate of venous ultrasonography. We evaluated the ratio of completed venous ultrasonography to completed D-dimer tests during the study period (

). The baseline ratios of completed venous ultrasonography to completed D-dimer tests for patients 65 years and older were 1.17 in intervention clinics and 1.55 in control clinics (ratio difference, −0.38; 95% CI, −0.63 to −0.11; 

 = .004). After alert activation, significant increases occurred in the ratios of completed venous ultrasonography to completed D-dimer tests for patients 65 years and older. Although the ratios increased in both clinic groups, the increase in intervention clinics was much greater. During the period when the alert was activated in intervention clinics and control clinics, the ratios of completed venous ultrasonography to completed D-dimer tests for patients 65 years and older were 3.85 in intervention clinics and 7.29 in control clinics (ratio difference, −3.44; 95% CI, −4.84 to −2.04; 

<.001). We saw a persistent ratio in intervention clinics when the alert was activated in both clinic groups (from 4.05 to 3.85; 95% CI, −0.83 to 0.45; 

= 0.57). The ratio in control clinics when the alert was active showed a significant increase (from 2.25 to 7.29; 95% CI, 3.74-6.35; 

<.001), similar to our observation when the alert was initially activated in intervention clinics. The increase in venous ultrasonography per D-dimer tests may have been driven by the large decrease in D-dimer tests in the intervention group (the denominator in this comparison). However, the increase in the ratio of completed venous ultrasonography to completed D-dimer tests was more than can be accounted for by just the decrease in D-dimer tests.

	The ratio of completed venous ultrasonography to completed D-dimer tests for patients younger than 65 years showed a slight increase after the alert was activated in intervention clinics, but decreased over time. A slight increase in the ratio was also observed after initiation of the alert in control clinics. No persistent trends were observed in intervention clinics or control clinics among either age group of patients for the rate of completed ultrasonography per 1000 patient visits (

). The overall few patients being evaluated for VTE among all patients having clinic visits may account for why we did not see a consistent change in the rate of venous ultrasonography per 1000 clinic visits.

	The rate comparisons in Table 1 provide useful information but do not directly test whether changes at intervention clinics were significantly different from those in control clinics, considering the difference in baseline rates between the clinics. To more rigorously answer this question, we used negative binomial models to test interactions of the intervention variable with time (

). These models focused on changes from baseline until alerts were operating in intervention clinics for patients 65 years and older. The use of D-dimer tests for patients 65 years and older decreased in intervention clinics and control clinics during periods when the alert was active in intervention clinics; however, the decrease was significantly lower for intervention clinics than for control clinics (

<.001). The results did not change after taking into account potential clustering of effects within facilities. The ratio of completed venous ultrasonography to completed D-dimer tests increased in intervention clinics and control clinics during the intervention period, but the increase was significantly higher for intervention clinics (

 = .006). The rate of completed D-dimer tests per 1000 visits among patients younger than 65 years did not change in intervention clinics or control clinics during the intervention period. However, for patients younger than 65 years, there was a slight increase in completed venous ultrasonography (Table 1, Figure 3, and Figure 4).

	In theory, alerts within an EMR may help decrease prescribing errors, increase rates of appropriate preventive screening and prevention interventions, and improve the delivery of evidence-based medical practice.

 Computerized alerts have also been effective in improving physician prescribing practices.

 However, the use of alerts in practice has not always proven to work as promised. Clinicians can experience alert fatigue, in which receipt of too many alerts becomes frustrating, leading to ignoring

	Our results show that a targeted alert designed to foster evidence-based guidance (as recommended by the American Academy of Family Physicians and the American College of Physicians) can reduce the use of D-dimer testing among patients 65 years and older.

 One reason that our alert mayhave been effective is that it provided alternative diagnostic

	suggestions, rather than just discouraging the use of the D-dimer test. In addition, the effectiveness of the alert persisted over time, suggesting that patient-specific alerts presented at an appropriate time at the point of care may foster improved adherence to evidence-based guidelines.

	Alert messages generated in an EMR guiding the use of D-dimer testing may not be generalizable to other tests for which alternative testing is not readily available. In addition, the utilization of alerts within an EMR used for documenting care within an integrated healthcare system may not be generalizable to nonintegrated health systems in which additional testing (ie, radiologic services) may not be readily available within the system. We were also limited in the data available from the EMR system database. We did not have access to the reasons why clinicians ordered imaging procedures. This precluded inclusion of chest computed tomography as an outcome measure, as this is used for many reasons other than evaluation of PE. However, lower extremity ultrasonography is almost exclusively ordered for the evaluation of DVT.

	Despite randomization, the baseline utilization of D-dimer testing differed between intervention clinics and control clinics, so there were likely unmeasured differences between clinics in terms of practice patterns, patient data, or clinician characteristics. We attempted to adjust for these differences with intervention variable-×-time analyses, but recognize that this might not fully adjust for unmeasured differences between study groups. We chose a cluster randomization model to limit the potential of interprovider contamination due to communication between providers in the same medical facility regarding the use of D-dimer testing. We wanted to limit the possibility of a scenario in which one provider gets the alert, another in close proximity does not get the alert, and they would talk about the alert message between themselves. Future studies may be designed to randomize at the provider level to confirm these results. In addition, Kaiser Permanente of Colorado has had a consistently stable Medicare population. We would not expect differences in the characteristics of this patient population to manifest during the course of this short study. We observed a trend toward decreased D-dimer testing for patients 65 years and older in all clinics. This observation may be because of continuing education materials and programs and other communications to providers informing them about poor performance of the D-dimer test among patients in this age group. We also noted an increase in the rate of D-dimer testing for patients younger than 65 years. This observation may have been an unintended consequence of the alert itself. That is, did the message that D-dimer testing may not be appropriate in the elderly lead clinicians to conclude that D-dimer is appropriate in the nonelderly (although the alert did not appear when ordering a D-dimer test for patients in the younger age group). This increase in D-dimer testing among the younger group may have been appropriate or inappropriate, but seems to have occurred in both clinic groups after the alert was in widespread use. This question remains for future studies and analyses.

	Further studies are needed to test the best use and structure of computerized alerts. We suggest that alerts should be focused on high-cost, high-variation, and high-frequency conditions to minimize the number of clinician workflow interruptions. Our study was not designed to compare alerts with and without information about alternative diagnostic and treatment strategies, although we believe that alerts should contain this information to direct clinicians toward evidence-based alternatives rather than simply provide “negative guidance.” As EMR technology improves, it might be possible in the future to tailor alert messages to clinicians based on their practice patterns, so that each clinician receives alerts only in areas where their individual practice could benefit. Systems might also be able to remove alerts for individual clinicians who have demonstrated persistent improvement in ordering practices. These approaches should mitigate the occurrence of alert overload and minimize the possibility that clinicians disregard alert messages.

A targeted computerized alert at the time of physician order entry reduced the use of D-dimer testing among patients 65 years and older.

Susan M. Shetterly, MS

Articles

Improvements in Access and Care Through the Affordable Care Act

Automated Phone and Mail Population Outreach to Promote Colorectal Cancer Screening

Computerized Alert Reduced D-Dimer Testing in the Elderly