The American Journal of Managed Care Special Issue: Health Information Technology
Electronic Alerts and Clinician Turnover: The Influence of User Acceptance
This study sought to examine the impact of user acceptance factors of electronic health record-based alert notification systems on the satisfaction, intentions to quit, and turnover of providers who used them. Contrary to existing theory (both the JDRM and the UTAUT), we found that monitoring/feedback on EASs practices, training on the use of EASs, and supportive norms about EAS had little impact on provider satisfac- tion. However, monitoring/feedback were associated with increased intention to quit.
Our results suggest that EASs, and by extension EHRs, could become catalysts for turnover, unless providers clearly understand their value to delivering high-quality care effectively and efficiently. As evidenced by the non-significant relationships between monitoring/feedback and provider satisfaction, as well as the nonsignificant relationship between training and both satisfaction and intention to quit, our data suggest that the aforementioned facilitating conditions may be insufficient to accomplish this goal, though we have no specific details in our data about the quality of the feedback or training. More importantly, when providers do not perceive the value of these electronic aids to their practice, they might become dissatisfied with their work environment, and potentially seek work elsewhere altogether.
EASs likely represent one of the most frustrating components of EHRs for providers54-56—compared with paper communication systems, they are perceived to “increase the number of work items, inflate the time to process each, and divert work previously done by office staff to them.”57 Other work has shown that providers perceive many of the alerts they receive to be unnecessary,58 and has documented variable physician ac- ceptance of features like computerized reminders and electronic alerts.59 Therefore, future work should target the problem from multiple angles, such as content and design of feedback, effectiveness of training, and social influence factors, in addition to already ongoing efforts to optimize EAS design, so that it is inherently perceived as valuable by providers. The United States already has a shortage of primary care providers,2 and research shows dissatisfied providers are both leaving primary care for other specialties and/or leaving medi- cine completely.3
Several possible reasons might exist to explain the positive effect of monitoring/feedback on intention to quit. First, participants might have reacted more strongly to the monitoring aspect than to the feedback aspect of this construct. Second, the nature of the feedback provided could minimize feedback’s impact on satisfaction. Feedback characteristics can have a significant impact on its effectiveness at changing cognitions and behavior.60,61 Our ongoing research in another domain has found that feedback is often delivered primarily via written reports providing only numeric scores without correct solution information62 (one of the most powerful single characteristics of feedback interventions).60,61 Third, both feedback delivery mechanisms and providers’ perceptions of being monitored constantly by the organization could have led to the observed result.
In contrast, PPOV showed a direct positive relation- ship to provider satisfaction (providers who perceived greater value in electronic notifications were more likely to be satisfied); a direct negative relationship to turnover (providers who perceived greater value in electronic notifications were less likely to quit); and an indirect link to intention to quit via provider satisfaction (providers who perceived greater value in alert notifications were more likely to be satisfied, and in turn less likely to express intentions to quit.) The relationship between provider satisfaction and intention to quit is not surprising, as it has been well documented in the literature.63,64 The more novel finding in this research is the direct, negative relationship between PPOV and turnover (ie, providers at facilities with higher provider turnover rates have lower perceptions of value for EASs). We are not aware of any studies directly linking these types of perceptions to actual turnover, particularly at the organizational level with a national sample as large as this one: 2590 respondents at 131 facilities. From a scientific perspective, this finding links the JDRM and UTAUT: if users do not perceive EASs to be of value, EASs are more likely to be considered a demand rather than a resource (and thereby less likely to be accepted), thus leading to increased turnover. From a practical perspective knowing that EASs have to be perceived as performance enhancing by physicians in order for them to not nega- tively affect turnover should signal facility leadership to take care regarding how such systems are designed, marketed within the facility, and supported.
In addition to this important finding, we are also not aware of any studies simultaneously examining the ef- fects of satisfaction, intention to quit, and turnover in the healthcare setting. Understanding the interrelationships among user acceptance of technological tools intended to help providers, factors that impact this acceptance, and provider outcomes can help the design and implementation of HIT tools with which providers will want to work.
The study was conducted within the VA system, representing one of the largest and most sophisticated healthcare systems in the United States. Hence, in the spirit of constructive replication and to enhance external validity, we recommend that our findings be replicated in subsequent studies at other facilities without centralized organizational structure. Nevertheless, alerting systems such as the one we studied are being increasingly used across commercial EHRs. Second, the structure of the archival turnover data obtained for this study limits turnover analysis over time and prevents the application of statistical techniques such as survival analysis that lead to the most informative results for turnover-type data. Third, our sample consisted of employees who were all using the same EAS-capable EHR, limiting our ability to generalize results to other commonly used EAS-capable EHR systems (eg, Epic Systems, Verona, Wisconsin) that are officially certified (by an Office of the National Coordinator for Health Information Technology-Authorized Testing and Certification Body). Hence, we recommend that future research focus on more heterogeneous samples, examining different types of EASs and EHRs. Finally, although this study identified a new, very specific source of dissatisfaction and potential turnover among providers, future studies should examine the incremental contribution of this source in the context of more traditional predictors of provider satisfaction such as supervisory relations, availability of resources, and work environment condi- tions.65,66 We further encourage future research to closely investigate how providers’ perceptions of EHR variables develop over time, and whether system characteristics or more distal factors (eg, supervisory behavior) impact these perceptions.
We conclude that designing and implementing EHR-based notification systems effectively may no longer simply be assumed to be an antecedent to efficiency, safety, or quality of care; how these systems are implemented, accepted, and used in real-world practice, as our research shows, might also impact provider satisfaction and retention. Given the recent HITECH stimulus and the new healthcare law, EHRs will be a reality nationwide in a few short years and will connect members of the healthcare team like never before. In fact, one reason for the heavy emphasis on EHR adoption is to improve communication. Depending on how the EHR is designed and implemented, it can become a source of competitive advantage (or turnover) for clinical practices. In addition, how an organization creates and manages its internal policies can make or break both the safety and efficiency of the clinicians’ work. As EHRs become more widespread and providers increasingly communicate clinical information through EASs, institutions should consider strategies to help providers perceive greater value in these vital clinical tools.
Author Affiliations: Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, TX; Baylor College of Medicine, Houston, TX (SJH, DE, HS); University of Houston, TX (CS); University of Texas Health Science Center at Houston, (DFS).
Source of Funding: This work was supported by the VA National Cen- ter of Patient Safety and partially supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, and the Center for Innovations in Quality, Effectiveness and Safety (#CIN 13-413).
Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.
Authorship Information: Concept and design (SJH, CS, HS, DFS); acquisition of data (SJH, CS, DE, DFS, HS); analysis and interpretation of data (SJH, CS, DE, DFS, HS); drafting of the manuscript (SJH, CS, DE, DFS, HS); critical revision of the manuscript for important intellectual content (SJH, CS, DFS, HS); statistical analysis (SJH, CS); obtaining funding (HS); administrative, technical or logistic support (SJH, CS, DE, DFS); and supervision (HS).
Address correspondence to: Sylvia J. Hysong, PhD, Center for Innovations in Quality, Effectiveness and Safety (152), Michael E. DeBakey VA Medical Center, 2002 Holcombe Blvd, Houston, TX 77030. E-mail: email@example.com.
1. Felice ME. Reflections on why pediatrics does not have a primary care physician shortage at present. J Pediatr. 2011;158:523-524.
2. Association of American Medical Colleges. Physician Shortages to Worsen Without Increases in Residency Training. 2010. https://www. aamc.org/download/15316
/data/physician_shortages_to_worsen_ without_increases_in_residency_tr.pdf. Published 2010. Accessed April 26, 2013.
3. Landon BE, Reschovsky JD, Pham HH, Blumenthal D. Leaving medicine: the consequences of physician dissatisfaction. Medical Care. 2006;44:234-242.
4. Rabinowitz HK, Diamond JJ, Markham FW, Paynter NP. Critical factors for designing programs to increase the supply and retention of rural primary care physicians. JAMA. 2001;286:1041-1048.
5. Blumenthal D. Stimulating the adoption of health information technology. N Engl J Med. 2009;360:1477-1479.
6. Singer S, Shortell SM. Implementing accountable care organizations: ten potential mistakes and how to learn from them. JAMA. 2011;306: 758-759.
7. Barnes KA, Kroening-Roche JC, Comfort BW.The developing vision of primary care. N Engl J Med. 2012;367:891-893.
8. Johns G.The psychology of lateness, absenteeism, and turnover. In: Anderson N, Ones DS, Sinangil HK, Viswesvaran C, eds. Handbook of Industrial and Organizational Psychology. London: Sage; 2001:232-252. 9. Menachemi N, PowersTL, Brooks RG.The role of information tech- nology usage in physician practice satisfaction. Health Care Manage Rev. 2009;34:364-371.
10. Sittig DF, Singh H. Rights and responsibilities of users of electronic health records. CMAJ. 2012;184:1479-1483.
11. Ryan AM, Bishop TF, Shih S, Casalino LP. Small Physician Practices In NewYork Needed Sustained HelpTo Realize Gains In Quality From Use Of Electronic Health Records. Health Affairs. 2013;32:53-62.
12. Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med. 2012;125:209-207.
13. Jha AK, Burke MF, DesRoches C et al. Progress toward meaningful use: hospitals’ adoption of electronic health records. Am J Manag Care. 2011;17:SP117-SP124.
14. McAlearney AS, Robbins J, Hirsch A, Jorina M, Harrop JP. Perceived efficiency impacts following electronic health record implementation: an exploratory study of an urban community health center network. Int J Med Inform. 2010;79:807-816.
15. Chaudhry B, Wang J, Wu S et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144:742-752.
16. Elder NC, Vonder MM, Cassedy A. The identification of medical errors by family physicians during outpatient visits. Ann Fam Med. 2004;2: 125-129.
17. Hickner J, Graham DG, Elder NC et al.Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network. Qual Saf Health Care. 2008;17:194-200.
18. Canon SJ, Purifoy JA, Heulitt GM et al. Results: Survey of pediatric urology electronic medical records-use and perspectives. J Urol. 2011; 186:1740-1744.
19. Delbanco T, Walker J, Bell SK et al. Inviting patients to read their doctors’ notes: a quasi-experimental study and a look ahead. Ann Intern Med. 2012;157:461-470.
20. Elder KT, Wiltshire JC, Rooks RN, Belue R, Gary LC. Health information technology and physician career satisfaction. Perspect Health Inf Manag. 2010;7.
21. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28:119-139.
22. Bitton A, Flier LA, Jha AK. Health information technology in the era of care delivery reform: to what end? JAMA. 2012;307:2593-2594.
23. Singh H,Thomas E, Mani S et al.Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are elec-
tronic medical records achieving their potential? Arch Intern Med. 2009;169:1578-1586.
24. Committee on Patient Safety and Health InformationTechnology, Board on Health Care Services. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: National Academies Press; 2011. RefType: Report
25. HHS. Health InformationTechnology: Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology. Federal Register. 2012;77:45 CFR Part 170 RIN 0991-AB82 p. 13845.
26. Best R, Hysong SJ, Moore FI, Pugh JA. Evidence based approaches to primary care staffing [final report]. 01-185. 10-7-2005. San Antonio, TX, Veterans Evidence-Based Research Dissemination and Implementation Center.
27. Hysong SJ, Best RG, Pugh JA, Moore FI. Are we underutilizing the talents of primary care personnel? a job analytic examination. Implementation Science. 2007;2:1-13.
28. Hysong SJ, Amspoker A, Khan M, Johnson K, Gribble G. VISN 6 Ambulatory Care System Redesign Improvement Capability Project- Evaluation. Final Report for Fiscal Year 2011 to the VA Mid Atlantic Health Care Network. September 21, 2011.
29. Harrison DA, Newman DA, Roth PL. How important are job attitudes? meta-analytic comparisons of integrative behavioral outcomes and time sequences. Acad Manag J. 2006;49:305-325.
30. Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record- based settings. JAMA Intern Med. 2013;1-3.
31. Hysong SJ, Sawhney M, Wilson L et al. Provider management strategies of abnormal test result alerts: a cognitive task analysis. J Am Med Inform Assoc. 2010;17:71-77.
32. Singh H, Spitzmueller C, Petersen N, Sawhney M, Sittig D. Socio-technical predictors of missed test results in EHR-based settings: a national survey of primary care practitioners. Arch Intern Med. In press.
33. Singh H, Spitzmueller C, Petersen NJ et al. Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc. 2012.
34. Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. The job demands-resources model of burnout. Journal of Applied Psychology. 2001;86:499-512.
35. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. Mis Q. 2003;27:425-478.
36. Hsiao C, Beatty PC, Hing ES, et al. Electronic medical record/electronic health record use by office-based physicians: United States, 2008 and preliminary 2009. December 1, 2009. Hyattsville, MD: Divi- sion of Health Care Statistics, National Center for Health Statistics.
37. Longman P. Best Care Anywhere, 3rd Edition: Why VA Health Care Would Work Better For Everyone. San Francisco: Berrett-Koehler Publishers; 2012.
38. Hynes DM, Whittier ER, Owens A. Health information technology and implementation science: partners in progress in the VHA. Med Care. 2013;51:S6-S12.
39. Fung CH, Woods JN, Asch SM, Glassman P, Doebbeling BN. Variation in implementation and use of computerized clinical reminders in an integrated healthcare system. Am J Manag Care. 2004;10:878-885.
40. Hysong SJ, Pugh JA, Best RG. Clinical practice guideline implementation patterns in VHA outpatient clinics. Health Serv Res. 2007;42:84-103.
41. Brown SH, Lincoln MJ, Groen PJ, Kolodner RM. VistA--U.S. Department of Veterans Affairs national-scale HIS. Int J Med Inform. 2003; 69:135-156.
42. Best RG, Hysong SJ, Pugh JA, Ghosh S, Moore FI.Task overlap among primary care team members: opportunity for system redesign? Journal of Healthcare Management. 2006;51:295-307.
43. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc. 2004;11:95-99.
44. Boohaker EA, Ward RE, Uman JE, McCarthy BD. Patient notification and follow-up of abnormal test results. A physician survey. Arch Intern Med. 1996;156:327-331.
45. Campbell EG, Regan S, Gruen RL et al. Professionalism in medicine: results of a national survey of physicians. Ann Intern Med. 2007;147: 795-802.
46. Cutler DM, Feldman NE, Horwitz JR. U.S. adoption of computerized physician order entry systems. Health Aff (Millwood). 2005;24: 1654-1663.
47. Jha AK, Ferris TG, Donelan K et al. How common are electronic health records in the United States? A summary of the evidence. HealthAff(Millwood).2006;25:w496-w507.
48. Lyons SS, Tripp-Reimer T, Sorofman BA et al. VA QUERI informatics paper: information technology for clinical guideline implementation: perceptions of multidisciplinary stakeholders. J Am Med Inform Assoc. 2005;12:64-71.
49. Poon EG, Gandhi TK, Sequist TD, Murff HJ, Karson AS, Bates DW. “I wish I had seen this test result earlier!”: dissatisfaction with test result management systems in primary care. Arch Intern Med. 2004;164: 2223-2228.
50. Wahls TL, Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract. 2007;8:32.
51. Cortese C, Quaglino G.The measurement of job satisfaction in organizations: a comparison between a facet scale and a single-item measure. Testing, Psychometrics, Methodology in Applied Psychology. 2006;13: 305-316.
52. James LR, Demaree RG, Wolf G. Estimating Within-Group Interrater Reliability With and Without Response Bias. Journal of Applied Psychology. 1984;69:85-98.
53. Amos [Version 17.0]. Chicago: SPSS; 2006.
54. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. Pre- scribers’ interactions with medication alerts at the point of prescribing: A multi-method, in situ investigation of the human-computer interaction. Int J Med Inform. 2012;81:232-243.
55. Saleem JJ, Russ AL, Justice CF et al. Exploring the persistence of paper with the electronic health record. Int J Med Inform. 2009;78: 618-628.
56. Saleem JJ, Russ AL, Neddo A, Blades PT, Doebbeling BN, Foresman BH. Paper persistence, workarounds, and communication break- downs in computerized consultation management. Int J Med Inform. 2011;80:466-479.
57. McDonald CJ, McDonald MH. Electronic medical records and preserving primary care physicians’ time: comment on “electronic health record-based messages to primary care providers”. Arch Intern Med. 2012;172:285-287.
58. Hysong SJ, Sawhney MK, Wilson L et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak. 2011;11:22.
59. Fung CH, Tsai JS, Lulejian A et al. An evaluation of the Veterans Health Administration’s clinical reminders system: a national survey of generalists. J Gen Intern Med. 2008;23:392-398.
60. Hysong SJ. Meta-analysis: audit & feedback features impact effectiveness on care quality. Medical Care. 2009;47:356-363.
61. Kluger AN, DeNisi A. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin. 1996;119:254-284.
62. Hysong SJ,Teal CR, Khan MJ, Haidet P. Improving quality of care through improved audit and feedback. Implement Sci. 2012;7:45.
63. Conway N, Briner RB. Full-time versus part-time employees: Understanding the links between work status, the psychological contract, and attitudes. Journal of Vocational Behavior. 2002;61:279-301.
64. Turnley WH, Feldman DC. Re-examining the effects of psychological contract violations: Unmet expectations and job dissatisfaction as mediators. J Organ Behav. 2000;21:25-42.
65. Hysong, SJ, Best RG, Bollinger M.The Impact ofVA’s Intramural Research Program on Physician Recruitment and Retention: annual meeting of the Veterans Administration Health Services Research & Development Program and Special Network Directors’ Poster Session of the 2007 Annual Meeting of the Veterans Administration Health Services Research and Development Service. 2007.
66. Buchbinder SB, Wilson M, Melick CF, Powe NR. Primary care physician job satisfaction and turnover. Am J Manag Care. 2001;7: 701-713.