https://www.ajmc.com/journals/issue/2014/2014-11-vol20-sp/electronic-alerts-and-clinician-turnover-the-influence-of-user-acceptance
Electronic Alerts and Clinician Turnover: The Influence of User Acceptance

Sylvia J. Hysong, PhD; Christiane Spitzmuller, PhD; Donna Espadas, BS; Dean F. Sittig, PhD; and Hardeep Singh, MD, MPH

Retaining primary care providers (PCPs) is critical to ensuring healthcare access and quality. However, PCPs are moving to other specialty areas or leaving medicine altogether, a significant threat to high-quality care in many US regions.1-4 Recent healthcare legislation and initiatives such as the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009, the Affordable Care Act of 2010, and the Patient-Centered Medical Home make explicit resource provisions such as training, additional staff, and resources to implement electronic health records (EHRs), all of which could make primary care more attractive.5-7 While PCPs’ decisions to seek alternate employment might be determined by a multitude of factors,8 provider dissatisfaction with the implementation and meaningful use of EHRs9 may pose unique retention challenges despite the HITECH Act’s strong incentives for their use.

Implementing a full-service EHR constitutes a major organizational intervention; it presents a significant change in clinician-to-clinician communication and in some instances can require additional skills beyond those that were needed in paper-based systems to deliver care of comparable quality.10,11 For example, providers spend an average of 49 minutes per day reviewing and responding to electronic alert notifications, yet nearly half of these notifications do not contain messages that providers perceive to contain “high value” information.12 This volume of information is significantly higher than what would be expected in a paper system (given the additional resources involved with physical mail and messaging), and thus requires a different work strategy for accurate and timely handling. In addition, research has also shown marked differences in workflow efficiency of paper-based versus EHR13-15; paper-based clinical information often gets “lost in the shuffle” and becomes untrackable.16,17 The needed changes in communication and work flow attributable to EHR use are significant enough that despite their benefits, 12% of pediatric urologists reported they would retire if EHR use were mandated.18

Recent research has demonstrated a relationship between the use of health information technology (HIT) and physician satisfaction,9 although different components
of HIT have different effects on physician satisfaction. For example, compared with traditional, paper-based forms of communication, PCPs who communicated electronically with patients and other providers, and who shared their visit notes electronically with patients, were more likely to report higher satisfaction levels19; in contrast, PCPs who wrote prescriptions electronically were less likely to report high satisfaction levels.20 Ensuring physicians’ satisfaction with their work is important because poor satisfaction often leads to several undesirable results, including turnover, mental health concerns (eg, anxiety, depression, burnout), poorer relationships with patients, and reduced quality of care.21

Within the Accountable Care Organization model, EHRs are expected to facilitate communication and coordination, especially in the outpatient setting.22 Increasingly, practices are relying on EHR-based alerting systems (EASs) within their EHR to track, route, and communicate clinical information such as test results.23 This electronic communication may occur through an “asynchronous” alert notification inbox, much like email, where the sender and recipient need not be simultaneously engaged. Notifications transmitted through these systems could include test results, referrals, status updates on patients, and other provider-to-provider communications. Although many commercial EHRs already feature EASs functionality for communication, its use is expected to grow. For example, results management is one of the core EHR functionalities24 and key criteria for achieving Stage 2 meaningful use25 due to its potential to reduce lag time in recognition and treatment of medical problems, reduce redundant testing, and improve appropriate and timely follow-up. Unlike other EHR components, how- ever, the impact of EASs on provider satisfaction and turnover is not well documented.

PCPs’ utilization of EHR-based EASs has become an integral job characteristic, especially because PCPs utilizing EASs tend to spend significant amounts of time interfacing with them.12,26-28 Consequently, problems related to EASs could potentially impact PCP job attitudes such as satisfaction and intentions to quit, both known ante- cedents of turnover.29 For example, our previous research suggests providers using EASs receive an average of 56 to 63 alert notifications per day12,22,30; in addition, providers do little to customize their alerts interface to optimize efficiency and effectiveness. Rather, they employ varying strategies for managing these notifications, with mixed success,31 leading to information overload.32

PCPs have requested new visualization tools such as color coding and advanced filtering to address some of these problems.33 Thus, without modifications, EASs use could lead to a combination of high-volume, low-value work that could function as a driver of turnover for PCPs, rather than a source of retention. As there is no prior research that explores the relationship between EHR variables and provider outcomes, our objective was to conduct an initial examination to identify how providers’ perceptions of the use of EASs may impact their satisfaction and intention to quit. This research could inform strategies about how those perceptions could be altered if needed, and can also serve as a stepping-stone for the integration of EHR research with turnover research.

METHODS

Conceptual Model

To answer our research questions, we took guidance from 2 theoretical models to ultimately derive the model depicted in Figure 1. The Job Demands Resource Model of burnout (JDRM)34 posits that job demands (ie, aspects of the job such as, potentially, the use of EASs) that require sustained physical or mental effort and lead to increased workload can lead to negative outcomes such as low satisfaction, intentions to quit one’s job, and eventually actual turnover. The model further proposes that job resources (ie, aspects of the job that are functional in achieving work goals) that reduce job demands, or stimulate personal growth and development, relate to positive attitudinal outcomes and lower levels of withdrawal.

With respect to EASs, it is unclear whether PCPs perceive EASs as a demand or a resource. The second model, Venkatesh’s Unified Theory of Acceptance and Use of Technology (UTAUT),35 sheds some light on what may drive this decision for providers. The UTAUT proposes several factors that could impact the aforementioned outcomes; among these are performance expectancy (the extent to which the user believes the system will help attain gains in performance); social influence (the extent to which the user perceives that important others, such as family and friends, believe the system should be used), and facilitating conditions (the extent to which the user perceives that technical and organizational resources exist to support system use). We thus examined 4 specific examples of the types of factors proposed by Venkatesh and their impact on physician satisfaction, intention to quit, and turnover: (1) EAS-supportive norms, such as the extent to which colleagues use and see value in the notifications (an example of social influence), (2) whether providers receive feedback about their use of EASs, (3) whether providers receive training on the use of notifica- tions (both examples of facilitating conditions), and (4) the perceived contribution of EASs to provider effectiveness (an example of performance expectancy, henceforth referred to as provider perceptions of value [PPOV]). Based on the JDRM and UTAUT, we hypothesized that each of the 4 factors will positively impact provider satisfaction, and inversely relate to intention to quit. Furthermore, intention to quit and provider satisfaction will significantly impact turnover.

Design
The present study is part of a larger cross-sectional Web-based survey of EAS practices conducted between June and November 2010 on a nationwide sample of PCPs practicing at Department of Veterans Affairs (VA) medical facilities.30,32,33

Setting
The VA is the largest integrated healthcare system in the United States and one of the most advanced in terms of fully functional EHR use,36 with EAS-capable EHRs having been in place at all medical facilities for almost a decade.37 In addition, the VA provides various national-level resources to support providers in their use of the Computerized Patient Records System (CPRS, the VA’s EHR), including nationally developed training modules, clinical application coordinators whose role is to assist clinicians in using CPRS, and a national performance measurement and quality reporting system. Despite these nationally available resources, however, previous research has shown considerable variability in the implementation of standardized national resources across VA facilities38 (eg, computerized clinical reminders)39 and in some cases this local variability has significantly impacted quality and performance.40 It is this local variation and degree of adaptation among facilities, despite the availability of national-level resources, which inspired the facility-level analyses in this study.

Electronic alert system features. CPRS features an inbox style electronic alert notification system (“View Alerts”).41 The View Alert window is displayed when a provider logs into CPRS, notifying the user of clinically significant events such as abnormal diagnostic test results (see Figure 2; a full taxonomy of the available alert/notification types within CPRS has been described elsewhere).12 The user can customize how alerts are displayed in several ways via features such as sorting and turning off nonmandatory notifications.31 Alerts stay in the View Alert window for a prespecified time or until the user specifically acknowledges the alert (ie, clicks the alert to read it). Thus, the View Alert system is a system of asynchronous communication used nationally in the VA to facilitate communication among multiple members of the patient’s care team. A core set of CPRS functionalities is determined at the national VA level. In addition, individual facilities have the flexibility to alter some of the CPRS settings. For instance, some facilities opt to have providers receive a larger number of relevant alerts, while other facilities alter settings so providers only view certain types of alerts that are considered “mandatory” at the institution level.

Participants and Procedure
Details of the survey’s development are reported elsewhere.32 In brief, using a nationwide VA administrative database (VA Primary Care Management Module), we invited all VA PCPs with a minimum practice panel size of 250 patients (N = 5001). Of 5001 PCPs invited, 2590 (51.8%) responded, representing data from 131 different VA facilities. Figure 3 displays the distribution of respondents within a facility across all 131 facilities. Respondents were 55.4% female, 31.1% nonwhite, and 31.5% nonphysician providers (eg, physician assistants, nurse practitioners); and 82.1% had 2 or more years in VA practice (Table 1). Within VA primary care, nonphysician providers behave largely as physicians do: They have their own patient panels, do at least 85% of the same work as physicians (differences being largely administrative),42 and use CPRS in largely the same way. Consequently, physicians and nonphysician providers were treated as a single population and identified as PCPs for study purposes.

Our study was reviewed and approved by our local institutional review board. Participants were recruited as follows: We first asked chiefs of primary care at each facility to email information about the project and the upcoming survey to the PCPs at their respective sites. We subsequently invited all participants via a personalized email from the study’s principal investigator; this e-mail described the study and provided a link to the Web-based survey. To increase response rates, invitation e-mails and subsequent reminders were followed by telephone attempts to reach nonrespondents.

Measures

Table 2 contains a list of constructs, construct definitions, sample items, and response scales used for the current study.

User Acceptance Factors. Measures for EAS Supportive Norms, Monitoring/Feedback and Training Infrastructure, and PPOV were developed specifically for this study based on a literature review43-50 and refined through pilot-testing with PCPs. Descriptive statistics, correlations, and reliability coefficients for these measures appear in Table 3. Details on the development of the survey instrument are reported elsewhere.33

Provider Satisfaction and Intention to Quit. Satisfaction and intention to quit were each measured via a single survey item, as suggested by Cortese and Quaglino.51

Turnover. Facility-level voluntary turnover rates for 2010 were obtained from the Veterans Health Administration Service Support Center Human Resources Cube, a large administrative database repository maintained centrally by the VA.

Data Analysis

Facility-Level Aggregation. Because our primary out-come, turnover, is a facility-level variable, it was necessary to aggregate all predictors (monitoring/feedback, supportive norms, PPOV, training, intention to quit, and provider satisfaction) to the facility level (n = 131) in order to successfully assess their impact on facility-level turnover. To test whether within-facilities responses were sufficiently homogeneous to justify aggregation, we calculated rwg, a measure of interrater agreement,52 for each relevant variable for each facility. Average rwg score was 0.71, suggesting sufficient agreement to warrant aggregation.

Model Test. We used structural equation modeling to test our hypothesized path model. Figure 1 presents the model tested; purple lines and green lines denote the initial model; red and green lines denote the final, best-fitting model. We additionally computed simple bivariate correlations to further explore the data. All analyses were conducted using SPSS Amos 17.53

RESULTS

Descriptive Statistics
Table 3 presents means, standard deviations, and correlations among study variables. Of note, bivariate correlations indicated a significant positive relationship between intention to quit and facility level turnover (r = 0.169, P <.05); and a significant negative relationship between provider satisfaction and facility-level turnover (r = –0.167, P <.05). Additionally, supportive norms and PPOV each correlated with provider satisfaction (r = 0.286, P <.01 and r = 0.495, P <.01, respectively) and intention to quit (r = –0.170, P <.05 and r = –0.383, P <.01, respectively), whereas monitoring/feedback correlated only with intention to quit (r = 0.185, P <.05). These significant correlations suggest testing a complete model is warranted.

Test of Hypothesized Model
Initial model fit. The hypothesized relationships (ie, each factor independently predicts provider satisfaction and intention to quit, both of which intercorrelate and in turn predict turnover, depicted in Figure 1 in the purple and green lines) resulted in poor fit (RMSEA = 0.21, PCLOSE <.001) when tested as a cohesive model. Of note, facility-level turnover was unrelated to provider satisfaction or intentions to quit.

Final model. An important feature of the original model is that the factors in the model were considered orthogonal, independent predictors of satisfaction, intention to quit, and turnover. Bivariate correlations, however, suggested this was an incorrect assumption. Consequently, based on the initial model results and the simple bivariate correlations, we trimmed unnecessary relationships from the model, and allowed the predictors to covary. The resulting model showed good fit (RMSEA = 0.04, PCLOSE = 0.47), and is presented in Figure 1 (depicted by the green and red lines). As can be seen from the figure and consistent with the bivariate correlations analyses, the 4 factors are significantly correlated, and thus cannot be treated as independent predictors of pro- vider satisfaction. After accounting for intercorrelations amongst the independent variables, monitoring/feed- back significantly predicted intention to quit (b = 0.30, P <.01), and PPOV predicted both provider satisfaction b = 0.58, P <.01) and facility level turnover (b = –0.19, P <.05), all without relying on either provider satisfaction or intention to quit as intermediary mechanisms. Of note, high levels of monitoring and feedback were associated with increased intentions to quit.

DISCUSSION

This study sought to examine the impact of user acceptance factors of electronic health record-based alert notification systems on the satisfaction, intentions to quit, and turnover of providers who used them. Contrary to existing theory (both the JDRM and the UTAUT), we found that monitoring/feedback on EASs practices, training on the use of EASs, and supportive norms about EAS had little impact on provider satisfac- tion. However, monitoring/feedback were associated with increased intention to quit.

Our results suggest that EASs, and by extension EHRs, could become catalysts for turnover, unless providers clearly understand their value to delivering high-quality care effectively and efficiently. As evidenced by the non-significant relationships between monitoring/feedback and provider satisfaction, as well as the nonsignificant relationship between training and both satisfaction and intention to quit, our data suggest that the aforementioned facilitating conditions may be insufficient to accomplish this goal, though we have no specific details in our data about the quality of the feedback or training. More importantly, when providers do not perceive the value of these electronic aids to their practice, they might become dissatisfied with their work environment, and potentially seek work elsewhere altogether.

EASs likely represent one of the most frustrating components of EHRs for providers54-56—compared with paper communication systems, they are perceived to “increase the number of work items, inflate the time to process each, and divert work previously done by office staff to them.”57 Other work has shown that providers perceive many of the alerts they receive to be unnecessary,58 and has documented variable physician ac- ceptance of features like computerized reminders and electronic alerts.59 Therefore, future work should target the problem from multiple angles, such as content and design of feedback, effectiveness of training, and social influence factors, in addition to already ongoing efforts to optimize EAS design, so that it is inherently perceived as valuable by providers. The United States already has a shortage of primary care providers,2 and research shows dissatisfied providers are both leaving primary care for other specialties and/or leaving medi- cine completely.3

Several possible reasons might exist to explain the positive effect of monitoring/feedback on intention to quit. First, participants might have reacted more strongly to the monitoring aspect than to the feedback aspect of this construct. Second, the nature of the feedback provided could minimize feedback’s impact on satisfaction. Feedback characteristics can have a significant impact on its effectiveness at changing cognitions and behavior.60,61 Our ongoing research in another domain has found that feedback is often delivered primarily via written reports providing only numeric scores without correct solution information62 (one of the most powerful single characteristics of feedback interventions).60,61 Third, both feedback delivery mechanisms and providers’ perceptions of being monitored constantly by the organization could have led to the observed result.

In contrast, PPOV showed a direct positive relation- ship to provider satisfaction (providers who perceived greater value in electronic notifications were more likely to be satisfied); a direct negative relationship to turnover (providers who perceived greater value in electronic notifications were less likely to quit); and an indirect link to intention to quit via provider satisfaction (providers who perceived greater value in alert notifications were more likely to be satisfied, and in turn less likely to express intentions to quit.) The relationship between provider satisfaction and intention to quit is not surprising, as it has been well documented in the literature.63,64 The more novel finding in this research is the direct, negative relationship between PPOV and turnover (ie, providers at facilities with higher provider turnover rates have lower perceptions of value for EASs). We are not aware of any studies directly linking these types of perceptions to actual turnover, particularly at the organizational level with a national sample as large as this one: 2590 respondents at 131 facilities. From a scientific perspective, this finding links the JDRM and UTAUT: if users do not perceive EASs to be of value, EASs are more likely to be considered a demand rather than a resource (and thereby less likely to be accepted), thus leading to increased turnover. From a practical perspective knowing that EASs have to be perceived as performance enhancing by physicians in order for them to not nega- tively affect turnover should signal facility leadership to take care regarding how such systems are designed, marketed within the facility, and supported.

In addition to this important finding, we are also not aware of any studies simultaneously examining the ef- fects of satisfaction, intention to quit, and turnover in the healthcare setting. Understanding the interrelationships among user acceptance of technological tools intended to help providers, factors that impact this acceptance, and provider outcomes can help the design and implementation of HIT tools with which providers will want to work.

Limitations

The study was conducted within the VA system, representing one of the largest and most sophisticated healthcare systems in the United States. Hence, in the spirit of constructive replication and to enhance external validity, we recommend that our findings be replicated in subsequent studies at other facilities without centralized organizational structure. Nevertheless, alerting systems such as the one we studied are being increasingly used across commercial EHRs. Second, the structure of the archival turnover data obtained for this study limits turnover analysis over time and prevents the application of statistical techniques such as survival analysis that lead to the most informative results for turnover-type data. Third, our sample consisted of employees who were all using the same EAS-capable EHR, limiting our ability to generalize results to other commonly used EAS-capable EHR systems (eg, Epic Systems, Verona, Wisconsin) that are officially certified (by an Office of the National Coordinator for Health Information Technology-Authorized Testing and Certification Body). Hence, we recommend that future research focus on more heterogeneous samples, examining different types of EASs and EHRs. Finally, although this study identified a new, very specific source of dissatisfaction and potential turnover among providers, future studies should examine the incremental contribution of this source in the context of more traditional predictors of provider satisfaction such as supervisory relations, availability of resources, and work environment condi- tions.65,66 We further encourage future research to closely investigate how providers’ perceptions of EHR variables develop over time, and whether system characteristics or more distal factors (eg, supervisory behavior) impact these perceptions.

CONCLUSIONS

We conclude that designing and implementing EHR-based notification systems effectively may no longer simply be assumed to be an antecedent to efficiency, safety, or quality of care; how these systems are implemented, accepted, and used in real-world practice, as our research shows, might also impact provider satisfaction and retention. Given the recent HITECH stimulus and the new healthcare law, EHRs will be a reality nationwide in a few short years and will connect members of the healthcare team like never before. In fact, one reason for the heavy emphasis on EHR adoption is to improve communication. Depending on how the EHR is designed and implemented, it can become a source of competitive advantage (or turnover) for clinical practices. In addition, how an organization creates and manages its internal policies can make or break both the safety and efficiency of the clinicians’ work. As EHRs become more widespread and providers increasingly communicate clinical information through EASs, institutions should consider strategies to help providers perceive greater value in these vital clinical tools.

Print | AJMC Printing...