Variation in Implementation and Use of Computerized Clinical Reminders in an Integrated Healthcare System

Published Online: November 01, 2004
Constance H. Fung, MD, MSHS; Juliet N. Woods, MS; Steven M. Asch, MD, MPH; Peter Glassman, MBBS, MSc; and Bradley N. Doebbeling, MD, MSc

Objectives: To identify patterns of use of computerized clinical reminders (CCRs) across an integrated healthcare system and describe institutional factors associated with their implementation.

Study Design: Cross-sectional study.

Methods: At a national electronic health record (EHR) meeting, we surveyed 261 participants from 104 Veterans Health Administration (VHA) healthcare facilities regarding the number and types of CCRs available at each facility. Potential explanatory measures included perceived utility and ease of use of CCRs, training and personnel support for computer use, EHR functionalities, and performance data feedback to providers at each facility.

Results: The number of conditions with CCRs in use at a facility ranged from 1 to 15; most reported implementation of reminders for 10 of the 15 conditions surveyed. The most commonly implemented CCRs, used in more than 85% of facilities, were for conditions with VHA national performance measures (eg, tobacco cessation, immunizations, diabetes mellitus). The least commonly implemented CCRs were for post-deployment health evaluation and management, medically unexplained symptoms, and erectile dysfunction. Facilities that had implemented greater numbers of clinical reminders had providers who reported greater ease of use and utility of the reminders (P = .01).

Conclusions: VHA facilities vary markedly in their implementation of CCRs. This effect may be partly explained by greater incorporation of clinical reminders for conditions with performance measures. Further study is needed to determine how to best implement clinical reminders and the institutional factors important in their use.

(Am J Manag Care. 2004;10(part 2):878-885)

Computerized clinical reminders (CCRs) have been widely publicized as potential tools for changing behavior1 and improving quality of care.2-4 They have been particularly effective in improving adherence with preventive care and screening guidelines,2,5 monitoring diabetes,6 and treating hypertension.7 However, factors such as workload, time, and perceived reduction of the quality of the provider-patient interaction may be barriers to effective use of computerized reminders.8

The Veterans Health Administration (VHA), the United States' largest integrated healthcare delivery system, has invested heavily in the informatics infrastructure necessary to support CCRs. It has been on the forefront of developing these tools and incorporating them into the computerized patient record system (CPRS), which is the VHA's electronic health record (EHR). The VHA developed CPRS, an application used throughout VHA facilities that enables clinicians to review and analyze patient data and supports clinical decision making.9

Computerized clinical reminders may be developed and distributed at the national, regional, or local level. Although VHA mandates use of a few clinical reminders such as assessing hepatitis C risk and possible sexual trauma during military service, most CCRs have been locally initiated.10 After reminders are created, they are tested and activated. Some reminders are applicable to all patients, while others apply to a particular group of patients. VHA facilities can generate clinical reminder reports, which provide information, for example, about the number of patients in a clinic with completed clinical reminders and the number of patients eligible for the reminder.

Although CCR technology is in widespread use in VHA, data regarding CCR patterns of use in VHA and institutional factors associated with their use have not been available. A description of the different types of reminders implemented in VHA may help other large organizations interested in promoting this technology. Identifying factors influencing widespread implementation of CCRs is a necessary step to promote their dissemination. Hence, our primary objective was to survey VHA facilities for variations in the numbers and types of CCRs used across VHA facilities nationally. Our secondary objective was to identify institutional factors associated with increased implementation of CCR technology.


Study Population and Data

We assessed CCR use across the VHA by distributing a survey instrument to participants at the national Camp CPRS meeting that took place in Georgia in May 2003. This meeting focused on the VHA's EHR and in 2003, included 1304 representatives from 136 of the 142 VHA medical facilities participating in the VHA's External Peer Review Program (EPRP). The representatives, who are nominated by their facilities, may be clinical staff (eg, physician, nurses, others), administrative personnel (eg, chief of staff), or informatics experts. Many attendees are opinion leaders at their facility and have extensive experience with local CPRS capabilities, either as users or developers of clinical applications for the EHR and its decision support tools.

We received responses from 261 participants (20%) representing 104 VHA facilities (76%). Nonrespondents were not tracked because only those who volunteered were given the survey instrument. Sixty-five of the 104 facilities that responded to our survey had more than 1 respondent.

The American Hospital Association database provided additional background information about VHA facilities. It provided information about geographic distribution of participating facilities, teaching status affiliation, and size of the facilities.

Questionnaire Development

After discussion with the VHA staff that oversee, develop, or perform research on clinical reminders, including members of the National Clinical Practice Guideline Council and members of its ad hoc Clinical Reminders Committee, we chose to evaluate facilities' use of 15 different types of CCRs. We selected reminders that represent a broad range of conditions clinicians might encounter in various clinic settings. Some reminders were for conditions that have VHA national performance measures (eg, addressing tobacco cessation), whereas others (eg, low back pain) were for conditions without VHA national performance measures.

The remainder of the 77 items in the survey instrument focused on institutional factors hypothesized from our ongoing studies to be important in implementation of clinical guidelines11 or CCRs. The instrument assessed different forms of computer use (provider education, performance feedback, and clinical support such as the ability to retrieve radiological images), perceived utility and ease of use of CCRs, adequacy of computer training and CCR training, organizational support, hospital culture/climate, and availability of feedback mechanisms for modifying CCRs. Responses for this portion of the survey instrument were either dichotomous (yes or no) or measured on a 5-point Likert-type scale. Respondents who answered "don't know" were recoded as "missing" or "no." The questionnaire is available, upon request, from Dr. Doebbeling.

The University of Iowa/Iowa City VA institutional review boards approved the study protocol. The data management plan underwent subsequent review and approval from the institutional review board of the VA Greater Los Angeles Healthcare System.

Outcome Measures

Outcome measures were obtained from questionnaire responses. For these measures, we aggregated individual responses at the facility level. Although 39 facilities (37.5%) had only a single respondent, the other facilities had multiple respondents, including 29 facilities with 2 respondents, 23 with 3 respondents, 6 with 4 respondents, 1 with 5 respondents, 3 with 6 respondents, and 3 with 10 or more respondents. When more than 1 response per facility was available, we used mean scores for questions that had Likert-like response scales. For dichotomous variables, we assumed that respondents from any given facility, while knowledgeable about their home facility, would be unlikely to report seeing or using a clinical reminder that they have never seen or used. In contrast, it is possible that they may not have seen or used a clinical reminder that was available at their facility, but a colleague at their facility may have reported seeing or using the clinical reminder. Because our goal was to determine whether a clinical reminder exists or not at a given facility, we weighted "yes" responses more than "no" responses. We reasoned that the union rather than intersection of positive responses would more accurately reflect, for example, the actual number of conditions with clinical reminders in place at a particular facility. Therefore, we treated any positive answer among respondents at the facility as a true-positive facility response. For example, if there were 3 responses and 2 stated that CPRS was used for requesting consults and 1 did not (or vice versa), the facility score was based on a "yes" answer. In a sensitivity analysis, we weighted negative responses more heavily than positive responses.

The first outcome measure represents which CCRs were available at the facility. The second outcome measure, a "facility clinical reminder score" (minimum possible score = 0, maximum possible score = 15), was created by summing "yes" responses to questions surveying whether a facility had at least 1 clinical reminder for a particular health condition. This variable represented the level of implementation of CCR technology in VHA facilities. Although some conditions, such as diabetes, may have more than 1 clinical reminder (eg, glycosylated hemoglobin, microalbumin/creatinine ratio), we counted each condition only once in our facility clinical reminder score. We used t tests to determine whether responses to each question differed significantly for facilities with complete versus incomplete responses. There were 72 facilities with complete data and 32 facilities with incomplete data (missing data or "don't know" response) for the questions used to construct the facility clinical reminder score. The mean facility clinical reminder score was 9.1 (95% confidence interval [CI]: 8.6, 9.7) for facilities with complete responses versus 7.4 (95% CI: 6.3, 8.5) for those with incomplete responses (a difference of 1.7; 95% CI: 0.7, 2.8). Table 1 presents comparison data for VHA facilities that participated in Camp CPRS and had respondents to the survey versus VHA facilities that did not participate in Camp CPRS or did not have respondents to the survey. Facilities with complete and incomplete responses were similar with respect to number of acute-care beds, trainee and resident full-time equivalents, academic affiliation, and urban versus rural location (P > .05 for all comparisons).


Potential Explanatory Measures

PDF is available on the last page.
Adult ADHD Compendium
COPD Compendium
Dermatology Compendium
Diabetes Compendium
GI Compendium
Hematology Compendium
Immuno-oncology Compendium
Lipids Compendium
MACRA Compendium
Oncology Compendium
Pain Compendium
Reimbursement Compendium
Rheumatoid Arthritis Compendium
Know Your News
HF Compendium
Managed Care PODCAST