Rebecca G. Mishuris, MD, MS

To determine whether clinical decision support (CDS) is associated with improved quality indicators and whether disabling CDS negatively affects these.

Using the 2006-2009 National Ambulatory and National Hospital Ambulatory Medical Care Surveys, we performed logistic regression to analyze adult primary care visits for the association between the use of CDS (problem lists, preventive care reminders, lab results, lab range notifications, and drug-drug interaction warnings) and quality measures (blood pressure control, cancer screening, health education, influenza vaccination, and visits related to adverse drug events).

There were an estimated 900 million outpatient primary care visits to clinics with EHRs from 2006-2009; 97% involved CDS, 77% were missing at least 1 CDS, and 15% had at least 1 CDS disabled. The presence of CDS was associated with improved blood pressure control (86% vs 82%; OR 1.3; 95% CI, 1.1-1.5) and more visits not related to adverse drug events (99.9% vs 99.8%; OR 3.0; 95% CI, 1.3-7.3); these associations were also present when comparing practices with CDS against practices that had disabled CDS. Electronic problem lists were associated with increased odds of having a visit with controlled blood pressure (86% vs 80%; OR 1.4; 95% CI, 1.3-1.6). Lab result notification was associated with increased odds of ordering cancer screening (15% vs 10%; OR 1.5; 95% CI, 1.03-2.2).

The use of CDS was associated with improvement in some quality indicators. Not having at least 1 CDS was common; disabling CDS was infrequent. This suggests that meaningful use standards may improve national quality indicators and health outcomes, once fully implemented.

		The use of some electronic health record clinical decision support (CDS) was associated with improved quality of care among national primary care outpatient visits from 2006 to 2009.

		Most practices were missing at least 1 CDS module considered essential by meaningful use standards.

		Meaningful use standards may have a significant impact on national quality of care and health outcomes once fully implemented.

The CMS Electronic Health Record Incentive Program, better known as “meaningful use,” provides initial monetary incentives for adoption of electronic health records (EHRs) and subsequently imposes monetary penalties if physician practices do not meet requirements. The meaningful use program has spurred a significant rise in the number of office-based physicians reporting EHR use, from 24% in 2005 to 48% in 2009 to 72% in 2012.

The meaningful use goals and objectives are divided into 3 stages. Stage 1 requirements include using a certified EHR to capture and share patient data by 2015; stage 2, beginning in 2014, includes use of advanced care processes with clinical decision support (CDS); and stage 3, to begin in 2016, will require providers to demonstrate outcome improvements. The time frame for stages 2 and 3 was recently extended.

 As of March 2013, 73% of eligible providers had registered to participate in the meaningful use incentive program and 44% of eligible providers had received incentive payment for meeting meaningful use objectives.

Although the meaningful use requirements have already been established, the evidence is inconsistent regarding improvement in healthcare processes or patient outcomes as a result of the implementation of general and individual EHR components. Prior studies have shown that EHR-based CDS is associated with improved prescribing safety,

 CDS has also been associated with some improvements in quality indicators,

 Despite having EHRs, many physicians report being unable to complete basic panel management activities,

which affects their ability to deliver high-quality care for patients with chronic conditions.

Notwithstanding the mandate to use certified EHRs, use of an EHR in the outpatient setting has not yet been shown unequivocally to improve quality indicators across a broad range of practices and EHRs, especially because there is substantial variability among the 1600 certified EHR platforms.

In addition to the large number of EHRs, they also can be customized to each practice’s needs and resource availability, so results will vary between practices in terms of impact on care even with the use of a single vendor’s practice. Certain types of practices, such as smaller community-based practices, may be less able to fully implement a complete EHR with clinical decision support, which is the kind postulated to have the greatest potential to improve quality.

 Decisions about choices around the CDS functions implemented may affect the quality of care delivered by that practice. To further inform the potential effect of meaningful use stages 2 and 3, we examined the association between having CDS, not having CDS, and disabling CDS, and quality of care indicators using nationally representative data just prior to the start of the meaningful use incentive program.

We performed a retrospective, cross-sectional analysis of adult primary care ambulatory clinic visits in the National Ambulatory Medical Care Survey (NAMCS) and National Hospital Ambulatory Medical Care Survey (NHAMCS) outpatient department records from 2006-2009 in order to analyze data with little impact from current meaningful use incentive payments. We examined a number of patient and practice characteristics to determine any differences in access to practices with EHRs based on prior work showing associations between these characteristics and EHR availability.

On the premise that most preventive care takes place through a primary care practice, we limited our sample to just those practices.

The NAMCS and NHAMCS are conducted by the Ambulatory Care Statistics Branch of the National Center for Health Statistics (NCHS) of the CDC. These surveys focus on outpatient visits to non—federally funded US medical practices.

The NCHS uses a complex multistage probability sampling design based first on geography, then physician specialty, and finally patient visits to individual practices. Information is collected at ambulatory visits by participating practices or US Census Bureau representatives. The NCHS assigned weights to each visit record to allow for estimation of national statistics. The NCHS institutional review board approves the protocols for the NAMCS and NHAMCS, including a waiver of informed consent for patients.

The NAMCS collects information at a visit level from community-based physician practices; from 2006 to 2009, approximately 30,000 community-based office visits were recorded annually. The NHAMCS collects information at a visit level from hospital-based physician practices; from 2006 to 2009, approximately 34,000 hospital-based office visits were recorded annually.

In all years included in the analysis, both the NAMCS and NHAMCS included questions in their intake surveys that asked whether the practice had an electronic medical record (for other than billing purposes), a computerized patient problem list, computerized warnings of drug interactions or contraindications, a computerized system to view lab results, a computerized system that highlights out-of-range lab values, and a computerized system for reminders for guideline-based interventions and/or screening tests. Response variables indicated that a practice had a function (“yes”), did not have a function (“no”), had the function but disabled it (“turned off”), or was unsure if the practice had the function (“don’t know” or “blank”).

The NAMCS and NHAMCS record health information for each visit sampled. The respondent indicates the top 3 reasons for the visit, which are then coded by NCHS; the patient’s vital signs, including blood pressure; whether any health education was ordered or provided during the visit; and whether an influenza vaccination was ordered or administered during the visit. The respondent also indicates any screening services ordered or performed during the visit. The respondent can record mammography, Pap smears, and scope procedures (NCHS records and codifiers, free text scope procedures, 1 of which is sigmoidoscopy/ colonoscopy), among others.

The NAMCS and NHAMCS collect patient and practice demographic information: patient age, patient race, patient income category (imputed based on patient zip code by NCHS), primary reason for the visit, region of the country, practice ownership, and number of physicians in the practice (solo or non-solo). Race data are collected in the manner already in use by the practice. Due to high nonresponse rates for race (24% to 33% in NAMCS, and 12% to 15% in NHAMCS for the years included), the NCHS also provides a race variable imputed based first on the patient’s location, then on physician specialty and 

International Classification of Diseases, Ninth Revision, Clinical Modification

 code for primary diagnosis, and finally on a random basis.

Our predictors of interest were EHR CDS modules mandated by stage 1 of meaningful use: electronic problem lists, preventive care reminders, lab result reporting, out-of-range lab notification, and drug-drug interaction warnings. We performed analyses with each predictor separately, and performed analyses combining them into a composite group of key EHR features. Based on prior work that examined associations between certain EHR features and clinical outcomes,

 we developed a matrix between CDS functions and the clinical outcomes of interest and performed analyses accordingly (

Our main outcomes of interest were based on national quality metrics related to adult primary care: blood pressure control (defined as a systolic blood pressure less than 140 mm Hg), age- and gender-appropriate cancer screening (mammography, Pap smear, and sigmoidoscopy or colonoscopy), health education for particular conditions, influenza vaccination during the months of October through March, and adverse drug events (as measured by visits coded to be related to an adverse drug event). We also combined the receipt of cancer screening, health education, and influenza vaccination into a composite “preventive care” outcome measure.

We were interested in the difference in outcomes based on whether a practice had a CDS function or not. Specifically, we examined whether there were significant differences in the rates of the clinical outcomes depending on whether a visit was to a practice with the CDS or without the CDS.

We categorized practices into 3 groups: all CDS tools active; without 1 or more CDS function; or any disabled CDS. We considered the practice to have all CDS active if they answered “yes” to having all 5 of the individual CDS. We considered the practice to be without 1 or more CDS functions if they answered “no,” “turned off,” don’t know,” or “blank” to any of the CDS functions. We considered the practice to have “disabled CDS” if they answered “turned off” for any of the CDS functions.

We collapsed practice ownership into 5 categories: physician or physician group, community health center, health maintenance organization, medical or academic health center, and other. All other covariates were used as provided by NCHS.

The unit of analysis was the patient visit. We followed NCHS guidelines based on the hierarchical sampling design. We performed all statistical tests on data with less than a 30% relative standard error (the standard error divided by the estimate itself) and more than 30 sample records.

We used multivariable logistic regression modeling to control for a number of patient and practice level covariates. We modeled age as a continuous variable, and all other covariates as categorical variables. We evaluated categorical variables with the χ2 test and continuous variables with the t test. We performed multivariable logistic regression modeling for each quality measure individually. The main predictors of interest were the status of the EHR CDS function. We considered P values of less than .05 to be significant.

We used SAS statistical software (version 9.3; SAS Institute, Cary, North Carolina) for all analyses.

From 2006-2009, the NAMCS/NHAMCS databases included 104,102 adult primary care visits, representing 2 billion adult primary care visits in the United States. Of these, 45% of visits were to practices with EHRs. Of those visits, 97% also had at least 1 of the 5 CDS functions of interest. Of visits using EHRs with at least 1 CDS function, 77% did not have (for any reason), and 15% had disabled (actively turned off), at least 1 of the 5 CDS functions of interest.

We compared patient and practice demographics for visits to clinics with all the CDS functions versus visits to clinics with at least 1 CDS function missing (

). Patients were of similar age (50.9 years vs 51.1 years,

 = 0.84). Rates of visits for flares of chronic problems (7%) and pre- or post surgery (2%) were similar between the 2 groups; rates of visits for new problems (39% vs 41%) and follow-up of chronic problems (32% vs 33%) were less common and visits for preventive care (17% vs 16%) were more common at clinics without at least 1 of the CDS functions versus clinics with all the CDS functions (

 <.0001). Of visits to practices with at least 1 CDS function missing versus those to practices with all the CDS functions, rates of visits to health maintenance organizations (3% vs 7%) were lower, and to physician groups (60% vs 50%), community health centers (5% vs 3%), or medical/academic health centers (18% vs 12%) were higher (

 <.0001). There were no differences in the distribution of visits to clinics without at least 1 of the CDS functions versus visits to clinics with all the CDS functions by gender, race, or region of the country.

Most patients who made visits (81%) had controlled blood pressure. Of eligible visits, there were low documented rates of age-appropriate cancer screening (13%) and influenza vaccination (4%). Approximately 0.2% of visits were related to an adverse drug event. Patients who made visits to clinics with all the CDS functions present versus those who visited clinics that were missing at least 1 of the CDS functions were more likely to have controlled blood pressure (86% vs 82%; OR 1.3; 95% CI, 1.1-1.5) and more likely to not have adverse drug event visits (99.9% vs 99.8%; OR 3.0; 95% CI, 1.3-7.3) (

We found similar results when specifically analyzing visits to clinics with all the CDS functions versus visits to clinics with any of the CDS functions disabled. Patients were more likely to have controlled blood pressure (86% vs 83%; OR 1.5; 95% CI, 1.02-2.3) and more likely to not have adverse drug event visits (99.9% vs 99.8%; OR 5.2; 95% CI, 1.4-19.0) (

We next examined only those CDS functions directly related to each clinical outcome. There were higher odds of receiving cancer screening if the visit was to a clinic with electronic lab results (15% vs 10%; OR 1.5; 95% CI, 1.03-2.2) or notification of acceptable lab ranges (16% vs 11%; OR 1.4; 95% CI, 1.03-1.8) than if the visit was to a clinic without these CDS functions. There were higher odds of having controlled blood pressure (86% vs 80%; OR 1.4; 95% CI, 1.3-1.6) if the visit was to a clinic with electronic problem lists rather than to a clinic without this EHR function (

Because of extremely small numbers and/or absence of outcomes in certain subgroups, we were unable to analyze the association of disabling particular CDS functions with specific clinical outcomes.

Using national surveys of ambulatory visits to primary care clinics, we found significant associations between the use of CDS and some, but not all, clinical quality measures before the enactment of meaningful use. Physicians reporting use of CDS had patients with better blood pressure control and fewer visits related to adverse drug events; these associations were present regardless of whether the physicians with CDS were compared with physicians without CDS or with physicians with disabled CDS. Physicians who reported having electronic problem lists had better rates of blood pressure control of their patients. Physicians with electronic lab results or notification of lab ranges were more likely to provide ageappropriate cancer screening. As we enter stages 2 and 3 of meaningful use in which clinicians must use CDS and then provide data about their performance with respect to patient outcomes, hopefully performance will continue to improve, though this remains to be demonstrated.

The incidence of not having at least 1 of the CDS functions was common, which is likely because the meaningful use mandate was not yet in place at the time of these national surveys. However, despite not having this mandate, the incidence of disabling any CDS was infrequent. Keeping decision support active is a “best practice” in clinical decision support,

but prior experience has demonstrated that many practices elected to disable specific types of decision support before meaningful use. These decisions can sometimes be indicated if the EHR is generating too many false positive warnings, which can create alert fatigue, although there typically are better solutions. The low rate of disabling CDS is reassuring and may indicate that the performance characteristics of CDS are improving. Notably, the odds of having controlled blood pressure and avoiding adverse drug events were higher at clinics with active CDS compared with those who had disabled it.

Prior studies of EHR use and clinical quality outcomes have been inconsistent in their findings. One study using earlier data from the same surveys used here found almost no associations between CDS use and clinical quality measures.

A more recent study found a significant, although small, improvement in blood pressure control of patients visiting physicians with CDS functions in their EHR versus patients visiting physicians without EHRs or CDS.

 In our study, we found an association between general CDS use and blood pressure control, and more consistent associations between quality indicators and CDS functions that were clinically related to these measures. These associations may be present because EHRs and CDS are evolving in a positive direction, spurred in part by the requirements of certification and also those of meaningful use.

Despite the positive findings, we have also found a lack of association between general CDS use and cancer screening, influenza vaccination, or health education, just as many previously have shown a lack of association between general EHR use and certain health outcomes.

 These findings may be because it is harder to change outcomes than quality processes. Some diseases, such as diabetes and hypertension, may be more amenable to clinical decision support than others, in part because there are clear guidelines and metrics.

This study has several limitations. We could not directly measure actual use of CDS during a visit, as the survey consisted of self-reported metrics. Even when physician practices have EHR features at their disposal, clinicians find ways to bypass or ignore them.20,21 Of the small number of practices that chose to disable CDS functionality, we do not know the reasoning behind their decisions, which may, in some cases, have been justified. In addition, the distinction between not having CDS functions and having turned off CDS may be sufficiently nuanced as to lead survey respondents to misclassify their EHR, although this would not affect our analysis of the difference between visits to clinics with all 5 of the CDS functions versus those without CDS. In addition, there were low rates of cancer screening and influenza vaccination captured in this visit-based incident data. There were also low rates of adverse drug event outcomes.

In contrast with some other studies, we found that the use of CDS functions was associated with improvements in measures of health quality at a national level. In addition, decisions to disable these CDS functions, although infrequent, negated these increases in care quality. With the continued implementation of meaningful use, there should be a further decline in practices without EHRs or that choose not to implement or to disable CDS. Overall, meaningful use standards that include CDS appear to have a significant positive effect on some national quality-of-care indicators and health outcomes. It will be important to evaluate the evolving impact of meaningful use as the stages continue to be more widely implemented and better integrated with care processes; we anticipate further gains in healthcare quality indicators and outcomes as a result.

 Division of General Medicine and Primary Care, Brigham Women's Hospital (RGM, JAL, DWB, AB); Brigham and Women’s Hospital (RGM, JAL, AB); Harvard Medical School (RGM, JAL, DWB, AB); Harvard Medical School Center for Primary Care (AB); Harvard School of Public Health (DWB), Boston, MA.

 Dr Mishuris is supported by an Institutional National Research Service Award to the Harvard Medical School Fellowship in General Medicine and Primary Care (T32 HP10251). No sponsor had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review or approval of the manuscript; or decision to submit the manuscript for publication.

Dr Bates holds a minority equity position in the privately held company Medicalis, which develops Web-based decision support for radiology test ordering, and he has served as a consultant to Medicalis. Dr Bates serves as an advisor to Calgary Scientific, which makes technologies that enable mobility within electronic health records. He is on the clinical advisory board for Patient Safety Systems, which provides a set of approaches to help hospitals improve safety. He has received funding support from the Massachusetts Technology Consortium. Dr Bitton serves part-time as a senior advisor to the Center for Medicare and Medicaid Innovation (CMMI). The research presented in this paper was conducted solely within the scope of his faculty appointments and research at Brigham and Women’s Hospital and Harvard Medical School, and in no way reflects any official position of CMMI. The authors report no other conflicts of interest.

 Concept and design (RGM, JAL, DWB, AB); acquisition of data (RGM, AB); analysis and interpretation of data (RGM, JAL, DWB, AB); drafting of the manuscript (RGM, JAL, AB); critical revision of the manuscript for important intellectual content (RGM, JAL, DWB, AB); statistical analysis (RGM, JAL, AB); administrative, technical, or logistic support (DWB); and supervision (JAL).

Articles

Using EHR clinical decision support is associated with improved quality of care. Most primary care practices are missing at least 1 "meaningful use" clinical decision support module.

Rebecca G. Mishuris, MD, MS

Articles

Using Electronic Health Record Clinical Decision Support Is Associated With Improved Quality of Care