Currently Viewing:
The American Journal of Managed Care January 2020
Using Applied Machine Learning to Predict Healthcare Utilization Based on Socioeconomic Determinants of Care
Soy Chen, MS; Danielle Bergman, BSN, RN; Kelly Miller, DNP, MPH, APRN, FNP-BC; Allison Kavanagh, MS; John Frownfelter, MD, MSIS; and John Showalter, MD
Eliminating Barriers to Virtual Care: Implementing Portable Medical Licensure
Pooja Chandrashekar, AB; and Sachin H. Jain, MD, MBA
Trust in Provider Care Teams and Health Information Technology–Mediated Communication
Minakshi Raj, MPH; Jodyn E. Platt, PhD, MPH; and Adam S. Wilk, PhD
The Health IT Special Issue: Enduring Barriers to Adoption and Innovative Predictive Methods
Ilana Graetz, PhD
What Accounts for the High Cost of Care? It’s the People: A Q&A With Eric Topol, MD
Interview by Allison Inserro
Does Machine Learning Improve Prediction of VA Primary Care Reliance?
Edwin S. Wong, PhD; Linnaea Schuttner, MD, MS; and Ashok Reddy, MD, MSc
Health Information Technology for Ambulatory Care in Health Systems
Yunfeng Shi, PhD; Alejandro Amill-Rosario, MPH; Robert S. Rudin, PhD; Shira H. Fischer, MD, PhD; Paul Shekelle, MD; Dennis Scanlon, PhD; and Cheryl L. Damberg, PhD
The Challenges of Consumerism for Primary Care Physicians
Timothy Hoff, PhD
Advancing the Learning Health System by Incorporating Social Determinants
Deepak Palakshappa, MD, MSHP; David P. Miller Jr, MD, MS; and Gary E. Rosenthal, MD
Predicting Hospitalizations From Electronic Health Record Data
Kyle Morawski, MD, MPH; Yoni Dvorkis, MPH; and Craig B. Monsen, MD, MS
Opt-In Consent Policies: Potential Barriers to Hospital Health Information Exchange
Nate C. Apathy, BS; and A. Jay Holmgren, MHI
Currently Reading
e-Consult Implementation Success: Lessons From 5 County-Based Delivery Systems
Margae Knox, MPH; Elizabeth J. Murphy, MD, DPhil; Timi Leslie, BS; Rachel Wick, MPH; and Delphine S. Tuot, MDCM, MAS

e-Consult Implementation Success: Lessons From 5 County-Based Delivery Systems

Margae Knox, MPH; Elizabeth J. Murphy, MD, DPhil; Timi Leslie, BS; Rachel Wick, MPH; and Delphine S. Tuot, MDCM, MAS
This case study of 5 county-based delivery systems finds that existing specialty care relationships and information technology integration are important differentiating factors for e-consult implementation success.
Unique to this study of publicly financed delivery systems, we identified that networks and communications, especially prior relationships between primary and specialty care, emerged as novel critical factors underlying successful implementation. System 1, which most robustly implemented a local e-consult program, had very strong existing relationships with an internal network of specialist providers as well as both the director of specialty care and the director of ambulatory care on the e-consult implementation leadership team. Similarly, in system 2, early-adopter specialties already commonly engaged in curbside consults, demonstrating existing communication among primary care and specialty care providers. In contrast, system 5 did not have an existing specialist network interested in responding to e-consults submitted by local primary care providers and was unable to launch an e-consult platform. Although using outside specialty consultants could be a workaround to counterbalance limited primary care–specialty care relationships, it appeared that the lack of preexisting relationships dissuaded engagement in a pilot e-consult program from clinicians as well as further investment in e-consult program implementation by executive leaders. Lee et al also emphasize the importance of the primary care–specialty care relationship: “This is all relational, right? We forget that when we build these tools,” they quote from a PCC interviewee.10 Our study extends the findings of Lee et al by highlighting that relationships are important not only for day-to-day operations but also from the early program planning and development stages. We speculate that the importance of these trusting relationships may play a larger role for providers who care for low-income vulnerable populations who often do not have other choices for healthcare delivery.

Our study also reinforces the role of information technology (IT) integration in facilitating access to information and knowledge. For example, system 3 explicitly built e-consult functionality into existing EHRs to avoid further fragmentation despite challenges with its EHR vendor. Other systems similarly worried that clinicians would not access e-consult services if they were implemented as a stand-alone platform amid multiple already-existing EHR systems. At the same time, system 2 demonstrated that an add-on platform can be feasible to expand e-consult across clinicians using alternative EHRs—an approach also taken at a large public healthcare delivery system not involved in this study.5 Thus, although integrated technology has previously been touted as an essential factor for implementation success across diverse healthcare systems,6 we demonstrate that it is a facilitator, albeit an important one, but not a determinant of implementation success among systems that may already experience fewer technology resources.

All county-based systems in our case study described substantial implementation challenges. For example, both salaried and fee-for-service specialists expressed dissatisfaction with organizational incentives. Salaried specialists wanted more dedicated time to complete e-consults, whereas fee-for-service specialists wanted higher reimbursement rates per e-consult completed. Regarding relative priority, implementation barriers included clinician resistance to e-consult workflow changes as well as low organizational priority amid other initiatives or leadership changes. Prior literature has noted sustainable financial models and new operational workflows as common challenges across delivery system types.6 Regarding available resources, systems 2 through 5 reported unstable or lacking project management support as a challenge that inhibited progress. Although this challenge is also found in prior literature,6 our evaluation suggests that project management may be a more common concern in safety net systems operating with limited resources.

Several factors ultimately contribute to the degree of implementation success. Among our county-based systems, strengths in some areas appeared able to compensate for weaknesses in others, allowing implementation to move forward. For example, the strong preexisting primary care–specialty care relationships in system 2 may have provided a glue for the less integrated add-on platform used to expand e-consult services. Although suboptimal factors were seen in all systems, systems 4 and 5 did not pursue implementation because of an insurmountable collection of climate and readiness factors, including weak or nonexistent specialist relationships on top of misaligned payment, lack of dedicated project management resources, and competing priorities.


Although this study builds on existing literature about e-consult implementation, some important study limitations exist. We interviewed between 1 and 3 informants per system, with an emphasis on perspectives from informatics and/or ambulatory care leaders who were responsible for e-consult planning and implementation. Thus, we potentially missed important viewpoints—for example, from specialist champions. In addition, platform data capture a common time frame (January-June 2018) to demonstrate e-consult uptake, but inference from these data may be limited because there was no common steady state of implementation. For example, systems 1 and 2 began implementation much earlier than systems 3 and 4, and system 1 completed implementation while systems 2 and 3 were in progress. Available platform data cannot determine eventual sustainability in systems 2 and 3 nor whether false starts in systems 4 and 5 may eventually yield success.

Platform data, including number of e-consults completed and number of specialties engaged, are valuable indicators of e-consult uptake but are not a definitive marker of implementation success. For example, system 1’s workflow uniquely includes all referrals, which explains why its percentage of comanagement is lowest despite its implementation being most robust. Multiple data inputs—quantitative and qualitative—are needed to understand implementation progress. Overall, interviews with greater depth or more precise metrics would bolster our findings. Nevertheless, our study greatly benefits from triangulating multiple sources for rich perspectives on e-consult implementation.


Our study uniquely contrasts local e-consult program implementation experiences across 5 county-based public delivery systems in California that serve similar populations and share common constraints, such as challenges in garnering administrative project management support. Features that differentiated successful implementation outcomes included (1) strong existing relationships between primary care and specialist clinicians and (2) IT integration between the EHR and e-consult systems. A strong foundation of local primary care and specialty care relationships appears to be a prerequisite for strong e-consult implementation, even when leveraging additional capacity from external specialists. Health system leaders should consider strengthening primary care–specialty care networks before or during e-consult implementation, as technology alone is unlikely to overcome existing fragmentation and does not act as a relationship-building bridge. Future health IT policy should also encourage EHR vendors to offer e-consult function integrations, similar to other Meaningful Use17 criteria. In parallel, healthcare systems can prioritize e-consult workflows that function within existing EHR communication systems to increase ease of access, even if integrated solutions require trade-offs such as greater project expense or longer implementation timelines.

Many stars must align for successful implementation of new healthcare IT systems like e-consult. Our examination finds that strong existing relationships between primary care and specialist clinicians, along with close e-consult and EHR platform integration, are prominent guiding lights to implement e-consult systems. Ultimately, these factors will also support the strong care coordination needed for timely access to the right care with better quality, value, and clinician and patient experience.

Author Affiliations: Center for Excellence in Primary Care (MK), Division of Endocrinology and Metabolism (EJM), Division of Nephrology (DST), and Center for Innovation in Access and Quality (DST), University of California, San Francisco, San Francisco, CA; BluePath Health (TL), Larkspur, CA; Blue Shield of California Foundation (RW), San Francisco, CA.

Source of Funding: Blue Shield of California Foundation, grant #128430A.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (MK, EJM, TL, RW, DST); acquisition of data (MK, RW, DST); analysis and interpretation of data (MK, EJM, RW, DST); drafting of the manuscript (MK, TL); critical revision of the manuscript for important intellectual content (MK, EJM, TL, DST); and obtaining funding (RW).

Address Correspondence to: Margae Knox, MPH, University of California, Berkeley, School of Public Health, 2121 Berkeley Way West, Room 5435, Berkeley, CA 94720. Email:

1. Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact of and satisfaction with a new eConsult service: a mixed methods study of primary care providers. J Am Board Fam Med. 2015;28(3):394-403. doi: 10.3122/jabfm.2015.03.140255.

2. Kwok J, Olayiwola JN, Knox M, Murphy EJ, Tuot DS. Electronic consultation system demonstrates educational benefit for primary care providers. J Telemed Telecare. 2018;24(7):465-472. doi: 10.1177/1357633X17711822.

3. Olayiwola JN, Potapov A, Gordon A, et al. Electronic consultation impact from the primary care clinician perspective: outcomes from a national sample. J Telemed Telecare. 2019;25(8):493-498. doi: 10.1177/1357633X18784416.

4. Chen AH, Kushel MB, Grumbach K, Yee HF Jr. A safety-net system gains efficiencies through ‘eReferrals’ to specialists. Health Aff (Millwood). 2010;29(5):969-971. doi: 10.1377/hlthaff.2010.0027.

5. Barnett ML, Yee HF Jr, Mehrotra A, Giboney P. Los Angeles safety-net program eConsult system was rapidly adopted and decreased wait times to see specialists. Health Aff (Millwood). 2017;36(3):492-499. doi: 10.1377/hlthaff.2016.1283.

6. Tuot DS, Leeds K, Murphy EJ, et al. Facilitators and barriers to implementing electronic referral and/or consultation systems: a qualitative study of 16 health organizations. BMC Health Serv Res. 2015;15:568. doi: 10.1186/s12913-015-1233-1.

7. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: early insights. Issue Brief (Commonw Fund). 2011;23:1-14.

8. Yin RK. Case Study Research: Design and Methods. 4th ed. Thousand Oaks, CA: Sage Publications Inc; 2009.

9. Deeds SA, Dowdell KJ, Chew LD, Ackerman SL. Implementing an opt-in eConsult program at seven academic medical centers: a qualitative analysis of primary care provider experiences. J Gen Intern Med. 2019;34(8):1427-1433. doi: 10.1007/s11606-019-05067-7.

10. Lee MS, Ray KN, Mehrotra A, Giboney P, Yee HF Jr, Barnett ML. Primary care practitioners’ perceptions of electronic consult systems: a qualitative analysis. JAMA Intern Med. 2018;178(6):782-789. doi: 10.1001/jamainternmed.2018.0738.

11. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50.

12. Tuot DS, Liddy C, Vimalananda VG, et al. Evaluating diverse electronic consultation programs with a common framework. BMC Health Serv Res. 2018;18(1):814. doi: 10.1186/s12913-018-3626-4.

13. Kim-Hwang JE, Chen AH, Bell DS, Guzman D, Yee HF Jr, Kushel MB. Evaluating electronic referrals for specialty care at a public hospital. J Gen Intern Med. 2010;25(10):1123-1128. doi: 10.1007/s11606-010-1402-1.

14. Liddy C, Keely E. Using the Quadruple Aim framework to measure impact of heath technology implementation: a case study of eConsult. J Am Board Fam Med. 2018;31(3):445-455. doi: 10.3122/jabfm.2018.03.170397.

15. Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614-619. doi: 10.1007/s11606-009-0955-3.

16. Liddy C, McKellips F, Armstrong CD, Afkham A, Fraser-Roberts L, Keely E. Improving access to specialists in remote communities: a cross-sectional study and cost analysis of the use of eConsult in Nunavut. Int J Circumpolar Health. 2017;76(1):1323493. doi: 10.1080/22423982.2017.1323493.

17. Meaningful use. website. Published October 22, 2019. Accessed December 4, 2019.
Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up