Margae Knox, MPH

 Electronic consultation, or e-consult, systems improve specialty care access by conveying specialist expertise to primary care clinicians (PCCs) without requiring specialist visits. Our study evaluates organizational factors for e-consult implementation across 5 publicly financed, county-based health systems in California. Each system serves 40,000 to 180,000 culturally and linguistically diverse patients across 4 to 19 primary care locations.

 We interviewed leaders whose systems received grant funding between 2015 and 2017 to plan and implement e-consult. Interviews discussed platform selection, electronic health record (EHR) compatibility, PCC and specialist opinions, and project governance. We also collected implementing systems’ platform operations metrics.

 Mixed methods, including semistructured interviews and quantitative platform metrics. Interviews were analyzed in alignment with the Consolidated Framework for Implementation Research inner setting domain.

 Three of the 5 systems successfully implemented e-consults. System 1 sustained implementation across 27 specialties, system 2 achieved fragmented implementation, and system 3 reported early-stage implementation. Existing PCC-specialist relationships emerged as the strongest facilitator. E-consult—EHR technology integration was also important, although an add-on platform enabled e-consult expansion in system 2. Although all systems faced challenges, such as project management resourcing, systems 4 and 5 abandoned implementation amid compound climate and readiness barriers.

 Successful e-consult implementations in public delivery systems leveraged (1) prior primary care and specialty care clinician relationships and (2) integrated EHR and e-consult platforms. This contrasts with common expectations that new technology will overcome care delivery gaps. Findings add to existing e-consult implementation literature that emphasizes reimbursement and leadership champions.

Electronic consultation, or e-consult, improves specialty care access and equips primary care clinicians to manage more complex cases by bridging specialty expertise and primary care delivery. Despite immense benefits, systems have varying success in implementing e-consult technology.

This case study of 5 county-based health systems finds that amid similar patient populations and resource constraints, systems with prior specialist relationships and more closely integrated information technology (IT) infrastructure established more robust e-consult implementation.

Other managed care systems can benefit from this takeaway by focusing on specialty relationship building and IT integration as prerequisites for e-consult system development.

Electronic consultation, or e-consult, systems link primary care clinicians (PCCs) with specialist expertise, supporting PCCs’ ability to manage complex conditions. PCCs using e-consults report both high satisfaction and high value for themselves and their patients.

 E-consults provide specialist guidance that can either eliminate the need for in-person specialist appointments or ensure that in-person appointments are more useful by identifying diagnostics or tests that patients should complete in advance. One e-consult platform found that approximately 1 in 4 e-consults in a national sample either avoided an unnecessary referral or avoided referral to the wrong specialty.

 Health systems similarly report reduced specialist visit wait times after implementing e-consult as a result of fewer unnecessary or inappropriate referrals.

 These outcomes are particularly salient for patients in public delivery systems, who disproportionately experience fragmented care and long wait times.

Past research identifies that specialty care access and appointment wait times commonly drive e-consult implementation. Facilitators include executive and clinician leadership. Common barriers include specialist reimbursement, technology funding, and administrative support.

 However, no studies have examined whether these factors similarly influence e-consult implementation across publicly financed, county-based health systems.

We explored e-consult implementation factors across 5 California county-based public health delivery systems. Each system serves 40,000 to 180,000 patients who are culturally and linguistically diverse. Each system provides primary and specialty care to predominantly publicly insured or uninsured patients at 4 to 19 primary care locations per system. Between 2015 and 2017, the Blue Shield of California Foundation solicited proposals from public hospital systems through competitive and by-invitation processes to advance e-consult adoption in the California safety net. Grant amounts and terms were designed to match stages of planning and implementation, from short-term feasibility grants ($50,000-$100,000) to multiyear implementation or spread grants ($250,000-$300,000). Foundation funding also supported access to technical assistance with e-consult implementation experts (eg, consultative phone calls, meetings, collaborative webinars). Funding recipients were expected to share operational metrics and provide leadership interviews.

We used a mixed-method case study approach with qualitative and quantitative data.

 We conducted semistructured interviews with executive leaders approximately 1 year after systems began implementation. Interviewees included chief medical informatics officers (systems 1, 3, and 4), a chief medical officer (system 4), medical directors of ambulatory care (systems 1, 2, 3, and 5), a medical director of community health partnerships (system 1), and project managers (system 2). In contrast with other studies that focus on end users,

 we focused on executive leaders with knowledge of the implementation trajectory from early-stage planning to launch and expansion. Interviews lasted 30 to 60 minutes and were audio recorded and comprehensively summarized by the study team. All interviewees provided informed consent. Interviewees did not receive individual compensation but represented health systems that received implementation grant support. We also analyzed standard operational metrics to differentiate those systems that successfully implemented e-consult from those that did not. The University of California, San Francisco Institutional Review Board reviewed and approved this project (IRB #14-15193).

]) explored e-consult platform selection criteria, electronic health record (EHR) compatibility, primary and specialty care clinician leader opinions, and project leadership and management. Two coauthors (M.K., D.S.T.) conducted thematic analysis using a preliminary set of analytic codes. Initial codes encompassed 6 essential implementation factors from e-consult implementation literature: executive leadership, clinical champions, efficient workflows, funding, EHR integration, and specialty access.

 Coding was refined during analysis based on interviewee responses. For example, interviewees rarely discussed “efficient workflows,” but a “resources to support operations” code was added because of repeated mention.

Project codes were subsequently mapped with the Consolidated Framework for Implementation Research (CFIR), a framework that unifies multiple implementation theories to guide evaluation across different studies and settings.

 CFIR authors recommend that “CFIR not be applied wholesale to every problem” but rather “those concepts that will be most fruitful to study.”

 Thus, we concentrated on the inner setting domain to identify features unique to each system that may have influenced implementation. In contrast, we expected that other domains, such as intervention characteristic and outer setting, would yield less variation because we were evaluating the same intervention (e-consult) across similar organizational settings (county-based delivery systems in California).

Quantitative, descriptive operations data for the period January to June 2018 were obtained from the 3 systems that successfully implemented e-consult programs. Data included number of specialties offering e-consults and volume of e-consults completed (to identify breadth and depth of the service), average specialist e-consult response time (a measure of specialty care access), and percentage of e-consult requests that were virtually comanaged versus converted to an in-person specialist visit (a measure of e-consult appropriateness and effectiveness to maintain patients in their primary care homes). These have been previously identified as important core measures of e-consult implementation to achieve cost-effective, high-quality specialty care delivery while maintaining high care team and patient satisfaction.

 We triangulated the operations data and leadership interviews to determine e-consult implementation success: “no, not implemented”; “no, limited pilot”; “yes, early implementation”; “yes, fragmented”; and “yes, sustained.”

Systems achieved varied success in their e-consult implementation journeys (

). System 1 achieved sustained implementation success, rolling out e-consults across all 27 specialties and completing more than 8000 e-consults. System 2 achieved fragmented implementation using both integration with an existing EHR platform as well as an external add-on platform for users on different EHRs. System 2 made e-consults available for 9 specialties and completed 880 e-consults. System 3 achieved early implementation by the end of data collection, with e-consults available for 13 specialties and 161 e-consults completed. System 4 piloted e-consults with 2 specialties but was unable to obtain meaningful volume or expand the e-consult program. System 5 chose not to implement e-consult beyond its planning grant phase. Across systems, all organizations faced challenges with consistent project management support and requests from specialists for greater financial or time recognition for e-consult services. Preexisting substantial relationships with specialist networks facilitated implementation, as did EHR compatibility.

System 1 includes 9 health centers providing primary care, urgent care, and specialty services, as well as a consortium of nonprofit community health centers with 19 primary care clinics. System 1 achieved sustained implementation success, rolling out e-consults across 3 to 4 specialties each quarter from March 2016 through February 2018. System 1 was unique compared with systems 2 and 3 because e-consults were required as part of the workflow for all specialty referrals. From January to June 2018, 8000 e-consults were completed, with about 23% virtually comanaged (ie, patients received care via their PCC without an in-person specialty visit). Prior to e-consult implementation, about 45% of referrals were lost or lapsed longer than 3 months, frustrating patients and their PCCs, and wait times for some specialist appointments were longer than 6 months. In contrast, specialists’ average response time to an e-consult was less than 24 hours, exceeding organizational goals.

Interviews revealed that system 1 specialists—already a part of an internal network within the health system&mdash;“really drove this initiative…and helped to identify roadblocks and work through them.” Setting aside time, including some clinical time, was essential to getting e-consults “off the ground.” In addition, clinicians in system 1 are salaried, requiring dedicated time to respond to e-consults.

E-consults were implemented by making changes within the existing EHR system. The implementation process included weekly leadership meetings with the chief medical informatics officer, the director of specialty care, the director of ambulatory care, and a leader from the community health center consortium. In addition, monthly meetings with a steering committee reviewed data, collaboratively decided when to extend e-consult to new specialties, and coordinated workflow and trainings. The ambulatory care director noted, “The real key is getting people together in person to discuss what’s happening with e-consult. What are the ongoing challenges? What are the successes?” Physician-to-physician communication was key, whereas communication from nonphysicians was less effective.

System 2 includes 5 hospitals and a network of neighborhood-based clinics, including a community health center network. System 2 began rollout in June 2016 and chose a hybrid implementation approach to incorporate both internal (salaried) specialists and external (contracted) specialists. Criteria for early-adopter specialties included a high volume of referrals, a long wait time for in-person specialty care visits, and already-occurring curbside consults. Overall, the system achieved sustained yet fragmented implementation. Implementation began among specialists already using an internal EHR platform, which generated 63% of the e-consult volume (553 of 880 e-consults between January and June 2018). The remaining 37% of e-consult volume (327 of 880) for January to June 2018 was generated from an external e-consult platform. The external platform was added about 1½ years after initial implementation to connect community health centers that rely on a different EHR and also to provide access to specialty expertise not readily available locally.

Of the 9 specialties that implemented e-consult, 3 specialties reported an average volume of greater than 10 e-consults per month: cardiology, endocrinology, and gastroenterology/hepatology. Cardiology received the most e-consult requests, 76% of which were comanaged without requiring an in-person specialist visit. Response times for cardiology ranged from an average of 6 hours in May to 6 days in June (3 days average for January-June 2018).

Although 1 interviewee indicated that e-consult implementation was a top priority for the chief medical officer, another interviewee cited lack of primary leadership focus amid other competing priorities. Both acknowledged leadership turnover as an implementation challenge. Many clinicians viewed e-consults as an additional uncompensated task because contracts had not changed to recognize dedicated time for e-consult completion.

System 3 provides care to more than 180,000 county residents at 8 clinic health centers and 2 regional medical centers, and it interfaces with 2 federally qualified health center networks. System 3 initiated implementation in February 2018 and reported 161 e-consults across 13 specialties as of June 2018. Average specialist response time was 12 days. About 41% of e-consults were comanaged without need for an in-person specialist visit. Uptake may actually be even greater because platform data may not capture e-consults occurring via older workflows. System 3 benefited from an experienced informatics leader guiding implementation. However, implementation also conflicted with the organization’s financial sustainability goals because e-consults are not financially recognized compared with in-person visits under current fee-for-service reimbursement structures. “We are more financially incentivized to open new clinics than build e-consult systems,” elaborated the ambulatory director.

Many specialists were supportive of, and even advocated for, program implementation. Yet specialists also desired greater compensation beyond the fee-for-service contract administrative review rate offered for each e-consult. As the ambulatory director, who met with each specialty reviewer during implementation, explained, “E-consults are no longer a simple review of a referral. Because it takes 15 to 20 minutes to write a whole consultation.” For some specialists, the relatively small compensation was a barrier to engaging.

System 3 deliberately built its e-consult program as a component of its existing EHR to avoid adding a secondary or tertiary platform. There nevertheless remains a sense that the technology is clumsy and still too cumbersome. Implementation leaders expressed frustration that the EHR vendor provided so little support and that there was no potential to integrate with third-party vendors.

System 4 provides care to more than 145,000 county residents at 6 locations. The main medical center includes primary care, specialty care, and hospital services. System 4 piloted e-consults with 1 primary care clinic (7 of 150 PCCs within the system) and specialists from 2 of 22 local specialty services. System 4 chose an external e-consult platform that could integrate with the 3 existing EHR systems, as leaders recognized that clinicians and staff would not likely use a fourth system for patient care. The selected e-consult vendor was chosen for its extensive experience with training users and developing workflow guidelines. However, contract negotiations to secure the vendor were slow, and specialists were difficult to engage. System 4 was ultimately unable to expand the pilot and faced a decreased appetite among executive leaders to further invest in potential e-consultative relationships.

System 5 serves more than 40,000 patients across 9 clinics, with 3 clinics located at a central medical center. Three PCC champions from 1 site agreed to pilot the program; however, the system was unable to engage local specialist clinicians. The selected e-consult platform was favored by external consultants and health plan stakeholders because it offered the potential for external specialists to respond to e-consults. However, the external platform did not integrate with the system’s EHRs. Despite strong executive leadership interest, system 5 did not pursue implementation amid weak clinician engagement.

Our analyses examine a spectrum of e-consult program implementation success among publicly financed, county-based health systems. Our study adds a unique contribution to the existing literature, which has described implementation in a single system

 To our knowledge, only 1 study explores implementation across organizationally similar health systems, a study of implementation in 7 academic medical settings.

To place our analysis in context, we discuss our results in relation to the CFIR’s inner setting domain. Salient CFIR inner setting concepts include networks and communication, implementation climate (including compatibility, relative priority, and organizational incentives), and implementation readiness (including available resources and access to information or knowledge).

All 5 systems in our study shared the ability to articulate compatibility with organizational goals, such as increasing operational efficiency and enhancing access to specialty care, decreasing wait times, and/or avoiding unnecessary specialist visits. This is consistent with findings of prior evaluations of e-consult implementation across diverse health delivery settings.

 Additional goals also consistent with prior literature included decreasing leakage to specialists in other systems (system 1) and capacity building for PCCs to manage more complex cases (system 4).

 System 5 cited connections to specialists in other regions across the United States, consistent with findings of a prior study that suggested higher odds of e-consult implementation in rural locations due to severely limited access to local specialists.

 Unique to this study of publicly financed delivery systems, we identified that networks and communications, especially prior relationships between primary and specialty care, emerged as novel critical factors underlying successful implementation. System 1, which most robustly implemented a local e-consult program, had very strong existing relationships with an internal network of specialist providers as well as both the director of specialty care and the director of ambulatory care on the e-consult implementation leadership team. Similarly, in system 2, early-adopter specialties already commonly engaged in curbside consults, demonstrating existing communication among primary care and specialty care providers. In contrast, system 5 did not have an existing specialist network interested in responding to e-consults submitted by local primary care providers and was unable to launch an e-consult platform. Although using outside specialty consultants could be a workaround to counterbalance limited primary care—specialty care relationships, it appeared that the lack of preexisting relationships dissuaded engagement in a pilot e-consult program from clinicians as well as further investment in e-consult program implementation by executive leaders. Lee et al also emphasize the importance of the primary care–specialty care relationship: “This is all relational, right? We forget that when we build these tools,” they quote from a PCC interviewee.

 Our study extends the findings of Lee et al by highlighting that relationships are important not only for day-to-day operations but also from the early program planning and development stages. We speculate that the importance of these trusting relationships may play a larger role for providers who care for low-income vulnerable populations who often do not have other choices for healthcare delivery.

Our study also reinforces the role of information technology (IT) integration in facilitating access to information and knowledge. For example, system 3 explicitly built e-consult functionality into existing EHRs to avoid further fragmentation despite challenges with its EHR vendor. Other systems similarly worried that clinicians would not access e-consult services if they were implemented as a stand-alone platform amid multiple already-existing EHR systems. At the same time, system 2 demonstrated that an add-on platform can be feasible to expand e-consult across clinicians using alternative EHRs—an approach also taken at a large public healthcare delivery system not involved in this study.

 Thus, although integrated technology has previously been touted as an essential factor for implementation success across diverse healthcare systems,

 we demonstrate that it is a facilitator, albeit an important one, but not a determinant of implementation success among systems that may already experience fewer technology resources.

All county-based systems in our case study described substantial implementation challenges. For example, both salaried and fee-for-service specialists expressed dissatisfaction with organizational incentives. Salaried specialists wanted more dedicated time to complete e-consults, whereas fee-for-service specialists wanted higher reimbursement rates per e-consult completed. Regarding relative priority, implementation barriers included clinician resistance to e-consult workflow changes as well as low organizational priority amid other initiatives or leadership changes. Prior literature has noted sustainable financial models and new operational workflows as common challenges across delivery system types.

 Regarding available resources, systems 2 through 5 reported unstable or lacking project management support as a challenge that inhibited progress. Although this challenge is also found in prior literature,

 our evaluation suggests that project management may be a more common concern in safety net systems operating with limited resources.

Several factors ultimately contribute to the degree of implementation success. Among our county-based systems, strengths in some areas appeared able to compensate for weaknesses in others, allowing implementation to move forward. For example, the strong preexisting primary care—specialty care relationships in system 2 may have provided a glue for the less integrated add-on platform used to expand e-consult services. Although suboptimal factors were seen in all systems, systems 4 and 5 did not pursue implementation because of an insurmountable collection of climate and readiness factors, including weak or nonexistent specialist relationships on top of misaligned payment, lack of dedicated project management resources, and competing priorities.

Although this study builds on existing literature about e-consult implementation, some important study limitations exist. We interviewed between 1 and 3 informants per system, with an emphasis on perspectives from informatics and/or ambulatory care leaders who were responsible for e-consult planning and implementation. Thus, we potentially missed important viewpoints—for example, from specialist champions. In addition, platform data capture a common time frame (January-June 2018) to demonstrate e-consult uptake, but inference from these data may be limited because there was no common steady state of implementation. For example, systems 1 and 2 began implementation much earlier than systems 3 and 4, and system 1 completed implementation while systems 2 and 3 were in progress. Available platform data cannot determine eventual sustainability in systems 2 and 3 nor whether false starts in systems 4 and 5 may eventually yield success.

Platform data, including number of e-consults completed and number of specialties engaged, are valuable indicators of e-consult uptake but are not a definitive marker of implementation success. For example, system 1’s workflow uniquely includes all referrals, which explains why its percentage of comanagement is lowest despite its implementation being most robust. Multiple data inputs—quantitative and qualitative&mdash;are needed to understand implementation progress. Overall, interviews with greater depth or more precise metrics would bolster our findings. Nevertheless, our study greatly benefits from triangulating multiple sources for rich perspectives on e-consult implementation.

Our study uniquely contrasts local e-consult program implementation experiences across 5 county-based public delivery systems in California that serve similar populations and share common constraints, such as challenges in garnering administrative project management support. Features that differentiated successful implementation outcomes included (1) strong existing relationships between primary care and specialist clinicians and (2) IT integration between the EHR and e-consult systems. A strong foundation of local primary care and specialty care relationships appears to be a prerequisite for strong e-consult implementation, even when leveraging additional capacity from external specialists. Health system leaders should consider strengthening primary care—specialty care networks before or during e-consult implementation, as technology alone is unlikely to overcome existing fragmentation and does not act as a relationship-building bridge. Future health IT policy should also encourage EHR vendors to offer e-consult function integrations, similar to other Meaningful Use

 criteria. In parallel, healthcare systems can prioritize e-consult workflows that function within existing EHR communication systems to increase ease of access, even if integrated solutions require trade-offs such as greater project expense or longer implementation timelines.

Many stars must align for successful implementation of new healthcare IT systems like e-consult. Our examination finds that strong existing relationships between primary care and specialist clinicians, along with close e-consult and EHR platform integration, are prominent guiding lights to implement e-consult systems. Ultimately, these factors will also support the strong care coordination needed for timely access to the right care with better quality, value, and clinician and patient experience.

 Center for Excellence in Primary Care (MK), Division of Endocrinology and Metabolism (EJM), Division of Nephrology (DST), and Center for Innovation in Access and Quality (DST), University of California, San Francisco, San Francisco, CA; BluePath Health (TL), Larkspur, CA; Blue Shield of California Foundation (RW), San Francisco, CA.

Articles

This case study of 5 county-based delivery systems finds that existing specialty care relationships and information technology integration are important differentiating factors for e-consult implementation success.

Margae Knox, MPH

Articles

e-Consult Implementation Success: Lessons From 5 County-Based Delivery Systems