Jennifer L. Hefner, PhD, MPH

To characterize factors influencing the development and sustainability of data sharing in the Mid-Ohio Farmacy (MOF), a produce referral program implemented in partnership between a community-based organization (the Mid-Ohio Food Collective [“Food Collective”]) and an academic medical center (The Ohio State University Wexner Medical Center [OSUWMC]).

We used an in-depth case study approach to identify challenges that arose during implementation of the MOF and related solutions via semistructured interviews with representatives of both organizations (May-September 2020).

Key informants from OSUWMC (n = 20) and the Food Collective (n = 11) were identified using a combination of purposive and convenience sampling; they included administrators, project champions, clinical providers, and food pantry representatives. Interview transcripts were coded using a deductive dominant approach guided by a logic model aimed at determining the resources and activities relevant to the development of the partnership.

Challenges of cross-sector data sharing fit into 3 themes: data sharing regulations, data exchange capabilities, and cross-sector data integration. Overcoming these challenges required creative workarounds—for example, linking patients across organizations was done via establishment of a unique, partnership-specific patient identifier, which was incorporated into the health system’s electronic health record for continuity.

 Our findings suggest that current regulatory frameworks are misspecified to the growing interest in cross-sector partnerships between health care and community-based organizations. Future efforts to support these relationships should consider clarifying rules around data sharing and increasing Medicaid support for nonmedical, health-related social needs.

We identified challenges to cross-sector data sharing and the approaches used to overcome these challenges in the Mid-Ohio Farmacy, a partnership to address food insecurity.

Cross-sector data sharing is a critical piece of forming successful partnerships between health care organizations and community-based organizations.

Through an in-depth case study of the Mid-Ohio Farmacy, we identified 3 key challenges:
(1) data sharing regulations, (2) data exchange capabilities, and (3) cross-sector data integration.

In addition to policy solutions that support cross-sector data sharing, overcoming these challenges requires both a high level of technical skills by partners and innovative approaches to integrating data from community-based organizations into clinical care.

Building a culture of health requires strengthening the alignment of the health care and social services sectors.

 Cross-sector partnerships can range from screening for social needs and risks and referral to community resources to full integration of a shared data set and colocation of health care and social services delivery. Such alignment can help to break down professional siloes that have perpetuated inequities in underserved  communities.

Successful cross-sector partnerships require 4, generally stepwise, components: refining a shared purpose, sharing data with all partners, ensuring long-term financing, and building a governance structure emphasizing representation.

 Notably, data sharing capacity is correlated with movement through this progression. However, partnership development can face significant barriers,

 in part because of a misalignment of goals related to different organizational structures and work cultures across sectors,

 as well as different regulatory frameworks. Further, the fee-for-service reimbursement model does not typically support payment for social needs, limiting incentives for health care organizations to invest in cross-sector partnerships with social service agencies.

Despite such barriers, there is growing momentum toward cross-sector collaboration and attention from foundations and government agencies

; however, the extant peer-reviewed literature lacks exploration of successful cross-sector partnerships and specific use cases of data sharing process development and documentation of challenges and potential solutions.

 Our case study of the Mid-Ohio Farmacy (MOF) aims to address this gap by presenting an example of health care and social services sector collaboration and highlighting lessons learned related to data sharing for service integration. The goal of the MOF is to improve health outcomes by addressing food insecurity for at-risk patients with a referral from their health care provider to access fresh produce via a local network of food pantries.

 The following analysis characterizes factors influencing the development and sustainability of the data sharing process that supports the MOF food referral program and presents implications for both the regulation of cross-sector data sharing and the development of future cross-sector alignments aimed at addressing social needs and social risks.

The Mid-Ohio Food Collective (“Food Collective”), a regional food bank, created the MOF to partner with the health care sector. Engaging with the health care sector aligns with the vision of the Food Collective to offer nutritious foods to community members in need. The MOF first began with a partnership between a federally qualified health center (FQHC) and the Food Collective. Current partners include 3 FQHCs, 2 free clinics, a Medicaid managed care plan, and an academic medical center: The Ohio State University Wexner Medical Center (OSUWMC). This in-depth case study

 is focused on the partnership between OSUWMC and the Food Collective, which began in 2019 and currently includes 8 primary care and specialty clinics. The partnership between OSUWMC and the Food Collective was initiated by a program champion who had helped establish the initial partnership and then subsequently transferred employment to OSUWMC. Their existing trust and working relationship with the Food Collective, combined with the knowledge that the Food Collective was seeking to expand partners, created an opportunity for OSUWMC to join the MOF. This opportunity was supported by broader efforts at OSUWMC to collaborate with community partners to address social needs.

To receive a MOF referral, a patient must have an affirmative response to a validated food insecurity screening tool

 embedded into OSUWMC’s Epic electronic health record (EHR) and have a qualifying chronic condition (ie, diabetes, hypertension, obesity). Qualifying and interested patients are asked to sign a release of information (ROI) if the visit is in person or to give verbal consent via telehealth. The ROI allows OSUWMC to transfer protected health information (PHI) about the referral to the Food Collective. The clinician enters a unique patient identifier (ie, “RxID”) into an EHR data field. Nightly, a report identifying all newly referred patients is uploaded to a secure file transfer protocol (sFTP) server where it can be accessed by the Food Collective.

Patients who consent to the ROI are handed or mailed information about the MOF and a card with a barcode that links to their RxID. This card associates patient utilization of the food pantry with the RxID already stored in the Food Collective’s data system (ie, FreshTrak). A monthly report of food pantry utilization by referred patients is then generated by the Food Collective, encrypted, and shared with OSUWMC via the sFTP server to enable tracking referrals, utilization, and health outcome evaluation.

We conducted one-on-one semistructured interviews with OSUWMC administrators and providers and Food Collective stakeholders regarding development and implementation of the MOF. For OSUWMC administrators, we used purposive sampling to recruit key informants (ie, MOF champions, data administrators, privacy officers, institutional leaders). For OSUWMC providers, we used a convenience sampling approach, sending a recruitment email to a listserve that included physicians, pharmacists, and nurse practitioners practicing in MOF-affiliated clinics. Working with MOF leadership, we used a purposive sampling approach to recruit Food Collective administrators and staff from affiliated food pantries.

The interview guide was developed to explore the resources and activities relevant to the MOF evaluation logic model constructs (ie, inputs, activities, outcomes, impacts).

 This guide included questions about expectations of and experience with the MOF, the implementation process, MOF-related workflow changes, data resources to support the program, and factors perceived as influencing MOF sustainability. Interviews were conducted between May and September 2020, audio recorded, and transcribed verbatim. Additional sources of data included nonparticipant observation of program meetings and reviewing MOF administrative documents. This study was approved by The Ohio State University Institutional Review Board, and verbal informed consent was obtained from all participants.

We used a deductive dominant approach to transcript coding that allows for the identification of emergent themes.

 Specifically, a preliminary codebook was developed based on the MOF evaluation logic model constructs. Initial code definitions were drafted by 3 coders (D.M.W., M.J.D., and J.A.G.) who reviewed a subset of transcripts. The 3 coders applied this initial codebook to common transcripts (n = 3) and collectively identified and defined emergent subcodes. The coding team then split, merged, and refined codes and their definitions iteratively via “second cycle coding” processes based on further transcript review until consensus was achieved and no further iterations to the codebook were proposed.

 The coding team worked collaboratively to apply the final codebook to all transcripts and met throughout the coding process to discuss emergent themes. NVivo software (QSR International) was used for coding.

A total of 31 interviews were conducted with OSUWMC administrators and providers (n = 20) and representatives of the Food Collective and their affiliated pantries (n = 11). Interview lengths ranged from 18 to 64 minutes (mean, 33 minutes). Analysis revealed data sharing as a critical element in the successful formation of a working partnership between OSUWMC and the Food Collective. However, interviewees described several challenges—spanning 3 key themes—related to establishing data sharing processes between stakeholders: (1) data sharing regulations, (2) data exchange capabilities, and (3) cross-sector data integration. Below, we characterize these challenges and the solutions conceived by the MOF partners to address them.

A challenge at the core of the MOF’s potential feasibility were the Health Insurance Portability and Accountability Act (HIPAA) policies dictating the exchange of PHI between covered entities (ie, OSUWMC) and noncovered entities (ie, the Food Collective and associated food pantries). Although HIPAA allows for data sharing of PHI between covered and noncovered entities for the purpose of coordinating patient care (eg, as in the MOF) through a business associate agreement (BAA), Food Collective interviewees expressed concerns regarding what specifically is allowable under HIPAA and whether partner food pantries could be HIPAA compliant, ultimately resulting in their reluctance to enter into a BAA.

These concerns were resolved by requiring patients to consent to the ROI (see 

). Despite this resolution, this hurdle resulted in a great degree of legal back-and-forth between entities—creating an initial barrier to the formation of the partnership. Moreover, the ROI necessitated a more detailed clinical workflow to enable receipt and documentation of the ROI and created an additional potential barrier to patient participation in the MOF, as patients may be reluctant to share their PHI with external entities.

A second critical aspect of this challenge was that OSUWMC and Food Collective administrators reported different interpretations of the specific data elements that constitute PHI, as described by a Food Collective representative: “The information that is transferred in order to make this referral occur is basic demographic information.…In our understanding of that, that is personally identifiable information. But there was no diagnostic information or anything like that being transferred. So, we were getting conflicting reports too, of, you know, what is personal identifiable information vs what is personal health information and what level of risk do people run in transferring that type of information.”

To help future cross-sector partners anticipate such issues, representatives from the Food Collective advocated for greater sharing of experiences with data exchange between community-based organizations and health care institutions through established professional networks, such as Feeding America.

 Nonetheless, Food Collective representatives felt that more information about regulatory issues would be beneficial: “When we did this health partnership toolkit, this was a [glaring] area to me that was missing. I really think that [Feeding America] should commit a little bit more of [their] Healthcare Partnership toolkit to the legal stuff because at that point in time I felt like that was all that I was doing—viewing legal documents.”

The data exchange between OSUWMC and the Food Collective was viewed as the defining aspect of the MOF, but several challenges had to be overcome (see 

). First, a mechanism was needed to match patients across institutions. The ROI permits sharing of demographic information but not the patient’s medical record number. This restriction required use of a unique identifier, the RxID, that can be stored in the EHR to enable linkage of clinical and food pantry utilization data from the Food Collective’s FreshTrak data collection and storage platform. Customizing an unassigned data entry field to store the RxID data element (ie, a smart data element in Epic’s EHR) was required to define the discrete data item. Developing this RxID and customizing the EHR necessitated a high level of technological expertise, and the flexibility of OSUWMC’s EHR (ie, Epic) was credited with enabling use of the RxID.

A second challenge was establishing a bidirectional workflow to share data. The developed approach includes daily reports of referrals automatically sent through sFTP from OSUWMC to the Food Collective and monthly reports of food pantry utilization sent back through sFTP from the Food Collective to OSUWMC. This process enables OSUWMC to analyze patient food pantry utilization in tandem with clinical outcomes for quality improvement and research purposes. OSUWMC stakeholders highlighted the ability to assess patient use of the MOF as a key facilitator of program sustainability. Notably, Food Collective representatives remarked that their technological capacity, including their data collection and storage infrastructure and ability to develop a protocol for linking patient referrals and MOF records, is rare for a food bank.

Although the issue of exchanging information between the partners was resolved using the RxID, other EHR-related challenges remain problematic. For instance, a representative from the Food Collective noted how EHRs do not allow for linkages at the household level: “In health care, your EHRs don’t have really any links to link those family members together, whereas in our site, in our systems, we do link individual people and roll them up into household units that they had self-identified. That’s a bit of a barrier or a challenge because you could receive 2 referrals for a food-insecure person, and one’s a husband, one’s a wife, or siblings, but they are in the same household.”

Moreover, patient food insecurity status is currently stored at the patient level, rather than the encounter level, in the EHR, which limits routine screening and inhibits tracking of food security fluctuations over time. Interviewees also noted that some food pantries had limited technological capabilities due to reliance on volunteers with limited technological literacy and unreliable Wi-Fi, which hampered data collection and sharing at the local pantry level.

Although establishing the data sharing infrastructure helped to support the operational aspects of the MOF, unresolved challenges exist related to integrating social services data into clinical care (ie, integrating FreshTrak data with the OSUWMC EHR data) (see 

). At the time of this study, patient-level food pantry utilization data were not yet communicated to referring providers, despite their expressed interest in these data. Questions also emerged about best practices to share these data so that they could be useful at the point of care.

Despite the absence of full integration of data at the point of care, the feedback of referral information can be used to encourage providers to refer eligible patients to the MOF, as described by one physician: “They would send out numbers of who had referred and numbers of people [who] are referred at some point.…It wasn’t a competition or anything; they were just like, ‘Hey, you guys are doing great, this is how many people have referred.’ And you could see by provider, and I was like, ‘Oh, those providers are really referring people, I should probably do this more,’ and that kind of made me realize how widely applicable the referral process would be and maybe start doing it a little more. So, that was just helpful to know what your colleagues are doing and how you can improve your practice.”

A key finding of our study is that HIPAA allowances for data exchange between covered and noncovered entities may not be articulated with sufficient clarity.

 For instance, HIPAA presently allows for health care providers to share PHI with social service agencies without patient authorization, yet most health care providers remain uncertain and cautious in doing so. Recently proposed modifications to HIPAA attempt to clarify and codify this allowance in an effort to encourage cross-sector alignment.

 Further, the recently passed CURES Act Final Rule restricts information blocking by health care organizations—or refusal to share patient information.

 If the HIPAA modifications are passed into law, they will enable social service organizations to file a formal complaint with the Office of the National Coordinator for Health Information Technology (ONC) if a health care organization engages in information blocking or requires unnecessary contracts. In practice, however, social service organizations may be reluctant to file a complaint against a voluntary partner, as it could damage the viability of the partnership. Instead, the potential for this complaint may serve as a lever for social service agencies to compel information sharing. These changes may facilitate cross-sector data sharing but could potentially create more opportunities for privacy breaches or unintended uses of PHI; thus, monitoring the impact of enhanced data sharing on privacy will remain important.

Incorporation of the food security screening tool within the EHR has been key to the MOF partnership’s success. As such, the MOF presents a use case not only in cross-sector data exchange, but also for the implementation of EHR-enabled social risk screening within a clinical workflow. The interviewees in our study noted some limitations to this aspect of the program, including the need for a high level of technical capabilities, as well as misspecified storage of the data at the patient rather than the encounter level. These findings advance the current literature on how to incorporate social risk data

 into the EHR and, if successful, how to effectively engage with this type of data at the point of care.

 This issue is notable given that ONC proposed including social risk (eg, housing, transportation, food insecurity) data elements in version 2 of the United States Core Data for Interoperability (USCDI) standards.

 USCDI standards dictate what data elements an EHR should be capable of capturing and exchanging. There is vast potential for integration of social risk data to enhance population health by permitting greater diagnostic precision, facilitating shared decision-making, and promoting prevention and referral activities.

 Greater evaluation of use cases, such as the MOF, is needed to inform the integration of social risk data into EHRs and to identify best practices.

Our findings also suggest that more work is needed to develop approaches for integrating data fully across sectors. Although the MOF partners successfully developed a unique patient identifier enabling longitudinal tracking of patients in aggregate, the partnership lacks a shared database, preventing more nuanced integration of data into patient care and programming. Recent national efforts have aimed to increase participation of community partners in cross-sector data sharing toward the development of shared databases. For instance, the Robert Wood Johnson Foundation Data Across Sectors for Health initiative and the Centers for Medicare and Medicaid Innovation Accountable Health Communities program aim to create greater linkages between health care and social services agencies.

 As with our efforts, data sharing has emerged as a key facet of the studied models,

 yet additional guidance on creating fully integrated, shared databases at the point of care is still needed to leverage cross-sector alignment toward health equity.

Our findings should be interpreted with important limitations in mind. We analyzed a limited sample of stakeholders within a single partnership. We chose to focus on OSUWMC’s experience given its stage in development of bidirectional data exchange with the Food Collective, whereas other partners in the MOF, including FQHCs and Medicaid managed care plans, are less far along in their implementation and have different goals and resources to support the development of the data exchange. Similarly, as noted in our findings, the Food Collective’s technological capacity may exceed that of most food banks. Thus, our findings may have limited generalizability to other organizations within or beyond the MOF. However, moving forward, Feeding America has agreed to take over the ongoing support and maintenance of the FreshTrak platform and plans to use it as the basis for a national-level platform that will be offered to every Feeding America food bank and more than 30,000 food pantries at no cost, potentially mitigating some of this concern. Finally, our analysis was focused on the issues that pertain to data sharing, and we did not examine other issues relevant to cross-sector partnership development, such as the role of trust, sustainability, or governance issues. These topics remain ripe for further exploration.

Collaborations between the health care and social service sectors have promising potential to advance health equity via integrated referral models to address social risk and social needs. Our case study of the MOF referral partnership between the Food Collective and OSUWMC provides insight into the challenges of and potential strategies for creating and sustaining cross-sector data exchange.

The authors of this manuscript are responsible for its content, including data analysis. The authors would like to thank Nicolette Coovert for their assistance with data collection and analysis.

 Department of Family and Community Medicine, College of Medicine, The Ohio State University (DMW, JLH, AC), Columbus, OH; The Center for the Advancement of Team Science, Analytics, and Systems Thinking (CATALYST), College of Medicine, The Ohio State University (DMW, MJD), Columbus, OH; Department of Health Services Management and Policy, College of Public Health, The Ohio State University (JLH), Columbus, OH; School of Health and Rehabilitation Sciences, College of Medicine and The John Glenn College of Public Affairs, The Ohio State University (JAG), Columbus, OH; Mid-Ohio Food Collective (AH), Grove City, OH; Division of Endocrinology, Diabetes and Metabolism, Department of Internal Medicine, College of Medicine, The Ohio State University (JJJ), Columbus, OH.

 This project was supported by a grant from The Ohio State University Department of Family and Community Medicine Crisafi-Monte Endowment Fund. The content is solely the responsibility of the authors and does not necessarily represent the official views of the sponsor.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (DMW, JLH, MJD, JAG, JJJ, AC); acquisition of data (DMW, MJD, JAG); analysis and interpretation of data (DMW, MJD, JAG, JJJ); drafting of the manuscript (DMW, JLH, MJD, JAG); critical revision of the manuscript for important intellectual content (DMW, JLH, MJD, JAG, AH, JJJ, AC); provision of patients or study materials (AH, AC); obtaining funding (DMW); administrative, technical, or logistic support (DMW, AC); and supervision (DMW).

 Daniel M. Walker, PhD, MPH, Department of Family and Community Medicine, College of Medicine, The Ohio State University, 460 Medical Center Dr, Ste 520, Columbus, OH 43210. Email: Daniel.Walker@osumc.edu.

1. Wojcik O, Miller CE, Plough AL. Aligning health and social systems to promote population health, well-being, and equity. 

. 2020;110(S2):S176-S177. doi:10.2105/AJPH.2020.305831

2. Braveman P, Arkin E, Orleans T, Proctor D, Plough A. What is health equity? Robert Wood Johnson Foundation. May 1, 2017. Accessed June 16, 2021. https://www.rwjf.org/en/library/research/2017/05/what-is-health-equity-.html

3. Gottlieb LM, DeSalvo K, Adler NE. Healthcare sector activities to identify and intervene on social risk: an introduction to the 

. 2019;57(6 suppl 1):S1-S5. doi:10.1016/j.amepre.2019.07.009

4. Kosel KC, Nash DB. Connected communities of care in times of crisis. 

NEJM Catalyst Innovations in Care Delivery

. August 24, 2020. Accessed April 13, 2021. https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0361

5. Wu AW, Weston CM, Ibe CA, et al. The Baltimore Community-Based Organizations Neighborhood Network: Enhancing Capacity Together (CONNECT) cluster RCT. 

. 2019;57(2):e31-e41. doi:10.1016/j.amepre.2019.03.013

6. Schmit C, Kelly K, Bernstein J. Cross sector data sharing: necessity, challenge, and hope. 

. 2019;47(suppl 2):83-86. doi:10.1177/1073110519857325

7. Rittenhouse D, Ament A, Shortell S. Accountable communities for health: data-sharing toolkit. Center for Healthcare Organizational Innovation Research. Accessed July 10, 2021. http://choir.berkeley.edu/wp-content/uploads/2019/05/CHOIR_dec2016_revised-1.pdf

8. Nguyen OK, Chan CV, Makam A, Stieglitz H, Amarasingham R. Envisioning a social-health information exchange as a platform to support a patient-centered medical neighborhood: a feasibility study. 

. 2015;30(1):60-67. doi:10.1007/s11606-014-2969-8

9. Laurent AA, Matheson A, Escudero K, Lazaga A. Linking health and housing data to create a sustainable cross-sector partnership. 

. 2020;110(suppl 2):S222-S224. doi:10.2105/AJPH.2020.305693

10. National Interoperability Collaborative. Partnerships, programs, and platforms: addressing social determinants of health through multi-sector data sharing. AcademyHealth. April 2019. Accessed April 14, 2021. https://academyhealth.org/sites/default/files/partnerships_programs_platforms_april2019.pdf

11. Fichtenberg C, Delva J, Minyard K, Gottlieb LM. Health and Human Services integration: generating sustained health and equity improvements. 

. 2020;39(4):567-573. doi:10.1377/hlthaff.2019.01594

12. Landers GM, Minyard KJ, Lanford D, Heishman H. A theory of change for aligning health care, public health, and social services in the time of COVID-19. 

. 2020;110(suppl 2):S178-S180. doi:10.2105/AJPH.2020.305821

13. O’Neil S, Staatz C, Hoe E, Goyal R, Ward E. Data across sectors for health initiative: formative evaluation report. Mathematica. September 30, 2019. Accessed April 14, 2021. https://www.mathematica.org/publications/data-across-sectors-for-health-initiative-formative-evaluation-report

14. Mongeon M, Levi J, Heinrich J. Elements of accountable communities for health: a review of the literature. National Academy of Medicine. November 6, 2017. Accessed July 20, 2021. https://nam.edu/wp-content/uploads/2017/11/Elements-of-Accountable-Communities-for-Health_updated.pdf

15. Cartier Y, Fichtenberg C, Gottlieb LM. Implementing community resource referral technology: facilitators and barriers described by early adopters. 

. 2020;39(4):662-669. doi:10.1377/hlthaff.2019.01588

16. Walker DM, DePuccio MJ, Hefner JL, et al. Utilization patterns of a food referral program: findings from the Mid-Ohio Farmacy. 

. 2021;34(6):1174-1182. doi:10.3122/jabfm.2021.06.210036

17. Yazan B. Three approaches to case study methods in education: Yin, Merriam, and Stake. 

. 2015;20(2):134-152. doi:10.46743/2160-3715/2015.2102

18. Hager ER, Quigg AM, Black MM, et al. Development and validity of a 2-item screen to identify families at risk for food insecurity. 

. 2010;126(1):e26-e32. doi:10.1542/peds.2009-3146

19. Cooksy LJ, Gill P, Kelly PA. The program logic model as an integrative framework for a multimethod evaluation. 

. 2001;24(2):119-128. doi:10.1016/S0149-7189(01)00003-9

20. Hayes H, Parchman ML, Howard R. A logic model framework for evaluation and planning in a primary care practice-based research network (PBRN). 

. 2011;24(5):576-582. doi:10.3122/jabfm.2011.05.110043

21. Armat MR, Assarroudi A, Rad M, Sharifi H, Heydari A. Inductive and deductive: ambiguous labels in qualitative content analysis. 

. 2018;23(1):219-221. doi:10.46743/2160-3715/2018.2872

The Coding Manual for Qualitative Researchers

23. Humana; Feeding America. Food insecurity and health: resource toolkit. Humana. Accessed April 20, 2021. https://populationhealth.humana.com/wp-content/uploads/2020/06/Food_Insecurity_Toolkit_2020_-Final.pdf

24. Petchel S, Gelmon S, Goldberg B. The organizational risks of cross-sector partnerships: a comparison of Health and Human Services perspectives. 

. 2020;39(4):574-581. doi:10.1377/hlthaff.2019.01553

25. Office for Civil Rights, Office of the Secretary, HHS. Proposed modifications to the HIPAA privacy rule to support, and remove barriers to, coordinated care and individual engagement. 

26. 21st Century Cures Act: interoperability, information blocking, and ONC Health IT Certification Program. Office of the National Coordinator for Health Information Technology. Accessed June 15, 2021. https://www.healthit.gov/curesrule/

27. Green K, Zook M. When talking about social determinants, precision matters. 

. October 29, 2019. Accessed June 26, 2021. https://www.healthaffairs.org/do/10.1377/hblog20191025.776011/full

28. Khanna N. Complexities in integrating social risk assessment into health care delivery. 

. 2020;33(2):179-181. doi:10.3122/jabfm.2020.02.200016

29. United States Core Data for Interoperability: version 2. Office of the National Coordinator for Health Information Technology. July 2021. Accessed July 27, 2021. https://www.healthit.gov/isa/united-states-core-data-interoperability-uscdi#uscdi-v2

30. Gottlieb L, Colvin JD, Fleegler E, Hessler D, Garg A, Adler N. Evaluating the Accountable Health Communities demonstration project. 

. 2017;32(3):345-349. doi:10.1007/s11606-016-3920-y

31. Alley DE, Asomugha CN, Conway PH, Sanghavi DM. Accountable Health Communities—addressing social needs through Medicare and Medicaid. 

. 2016;374(1):8-11. doi:10.1056/NEJMp1512532

32. RTI International. Accountable Health Communities (AHC) model evaluation: first evaluation report. Center for Medicare and Medicaid Innovation. December 2020. Accessed July 10, 2021. https://innovation.cms.gov/data-and-reports/2020/ahc-first-eval-rpt

Articles

The authors identified challenges to cross-sector data sharing and the approaches used to overcome these challenges in the Mid-Ohio Farmacy, a partnership to address food insecurity.

Approaches for Overcoming Barriers to Cross-Sector Data Sharing

Current models for patient risk prediction rely on practitioner expertise and domain knowledge. This study presents a deep learning model—a type of machine learning that does not require human inputs&mdash;to analyze complex clinical and financial data for population risk stratification.

 A comparative predictive analysis of deep learning versus other popular risk prediction modeling strategies using medical claims data from a cohort of 112,641 pediatric accountable care organization members.

 “Skip-Gram,” an unsupervised deep learning approach that uses neural networks for prediction modeling, used data from 2014 and 2015 to predict the risk of hospitalization in 2016. The area under the curve (AUC) of the deep learning model was compared with that of both the Clinical Classifications Software and the commercial DxCG Intelligence predictive risk models, each with and without demographic and utilization features. We then calculated costs for patients in the top 1% and 5% of hospitalization risk identified by each model.

 The deep learning model performed the best across 6 predictive models, with an AUC of 75.1%. The top 1% of members selected by the deep learning model had a combined healthcare cost $5 million higher than that of the group identified by the DxCG Intelligence model.

 The deep learning model outperforms the traditional risk models in prospective hospitalization prediction. Thus, deep learning may improve the ability of managed care organizations to perform predictive modeling of financial risk, in addition to improving the accuracy of risk stratification for population health management activities.

The present study benchmarked a new deep learning methodology for patient risk stratification using clinical and financial data for a small pediatric accountable care organization (ACO) data set. The predictive validity of the deep learning model was higher than that of other popular population risk prediction modeling strategies that, unlike deep learning, require practitioner expertise and domain knowledge. The deep learning model, although preliminary, may:

Enable population risk stratification without the investment of human resources that other modeling techniques require

Improve the ability of pediatric ACOs and others conducting population health management to perform predictive modeling of healthcare utilization

The current US healthcare climate is focused on delivery transformation and reform through value-based models that increasingly hold healthcare organizations accountable for population-based outcomes. The accountable care organization (ACO) is a popular type of value-based model that relies on patient risk stratification to identify high-risk patient populations for targeted care coordination and population health management activities.

 Risk stratification of patients using predictive regression models has long been an analytic challenge because of sparse, high-dimensional, and noisy data from insurance claims.

Currently, the state-of-the-art risk stratification models rely on groupers of diagnosis codes developed in the early 2000s, such as Diagnostic Cost Group (DCG) and its Medicare equivalent DxCG,

 Clinical Classifications Software (CCS),

 and the Johns Hopkins Adjusted Clinical Groups (ACG) model.

 These grouper models are important tools for conducting risk stratification in the field of managed care, including in quality measurement and reporting, risk adjustment, and reimbursement. A 2013 study by Haas and colleagues explored the predictive validity of 6 popular risk stratification modeling techniques involving predictive multivariate modeling of administrative data into categories or weighted scores to predict future healthcare utilization.

 Although the study found that each of the 6 models had at least fair predictive validity, the evaluation was centered on an adult population, as were most published studies of risk adjustment.

As researchers and actuarial experts pointed out, many traditional risk adjustment methodologies were developed using a standard population and hence optimized with greater emphasis on adults, potentially limiting their predictive power in pediatric populations.

 In an effort to develop pediatric-focused risk adjustment, researchers have noted that due to the persistence of healthcare costs among certain groups of high-cost children, pediatric risk models benefit from including patient-reported measures of health status.

 Specifically, among Medicaid patients, the predictive validity of pediatric cost prediction models can be improved by adding survey-based measures to models of administrative data

; however, this methodology requires the collection of a large volume of survey data from the target patient population.

Recently, deep learning has made the implementation of predictive analytics easier due to an unsupervised learning approach with automatic feature engineering. The deep learning method is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using multiple processing layers and connections. This approach allows the computer to learn complex concepts by building them out of simpler ones, identifying patterns and dependencies in the data. Outside the healthcare domain, deep learning and artificial intelligence have transformed our daily lives, with applications in face recognition, credit rating, and instant approval of life insurance.

 Within healthcare, deep learning is in use in a variety of health information technology contexts, including genomic analysis and biomedical image analysis.

 The application of deep learning to patient-level risk prediction is a new area of exploration. Early uses of deep learning in electronic health record data have shown promising results, including the use of recurrent neural networks to predict future diagnoses and medication orders

 and autoencoders to predict patients’ health status.

Deep learning can handle and leverage the complex relationships in large, sparse, high-dimensional, and noisy data, with no data supervision or labeling needed. Embedding is a popular technique in which all the input concepts (ie, diagnosis and procedure codes) from the data set are mapped to vectors of real numbers in a low-dimensional continuous space where the distance between medical concepts conveys similarity between concepts. A 2016 study by Miotto et al explored the use of deep learning to predict a patient’s future medical conditions.

 The authors’ work suggested that deep learning can replace hands-on algorithm creation.

In the present study, we further develop this deep learning approach through the hypothesis that if the future diagnosis of a patient can be predicted, then deep learning can also predict clinical and financial risks. More specifically,

the primary objective of our study is to develop a framework for pediatric patient risk prediction that can be used to improve patient risk stratification. If successful, the model derived from this study could enable healthcare organizations to identify high-risk patients for care management interventions and improve financial risk forecasting. Our specific aims are (1) to use a deep learning model to develop a predictive algorithm for future healthcare utilization from ACO data (including diagnosis, demographic, financial, and provider data) and (2) to assess the predictive validity of this model. To our knowledge, this is the first deep learning study to utilize complex clinical and financial data for population risk stratification. To demonstrate the feasibility of this approach, we benchmarked a deep learning method on a small pediatric data set. We present the results of this preliminary work and discuss the implications of this model for patient health, healthcare practice, and public health policy and management.

Partners for Kids (PFK) is an ACO affiliated with Nationwide Children’s Hospital in Columbus, Ohio. As one of the oldest and largest pediatric ACOs, PFK assumes full financial risk for a pediatric population of more than 330,000 Medicaid-qualified children in central and southeastern Ohio.

 In accordance with the Common Rule (45 CFR 46.102[f]) and the policies of Nationwide Children’s Institutional Review Board, this study used a limited data set and was not considered human subjects research; it was therefore not subject to institutional review board approval.

 an unsupervised learning approach that uses neural networks to learn the relationships among medical codes. Although Skip-Gram is only a 3-layer neural network, we refer to it as a “deep learning” algorithm following the work of Minarro-Giménez et al

 We grouped medical codes (diagnoses, procedures, and drugs) by month for the current year and projected each patient’s medical codes into latent vector space utilizing Skip-Gram to predict patient hospitalization in the subsequent year (

). This model can learn the semantic information of medical codes and forms a condensed representation (ie, vector) for each medical code. This allows us to better utilize the medical codes for predicting future hospitalization. For more details on our analytic approach, please see our forthcoming technical paper.

The full data set for this study consisted of medical claims from 303,404 unique PFK members who were eligible at the start of 2014. Only patients with continuous eligibility for the 3-year period from 2014 to 2016 were selected, resulting in a cohort of 112,641 members. We excluded the subpopulation of children who were on and off Medicaid, as we could not get the complete data of their costs and utilization during those gaps. We developed our predictive model using 2014-2015 data and validated it with 2015-2016 claims data (Figure). Data elements extracted include (1) patient demographic data (ie, sex, age, and zip code) and (2) utilization data (ie, medical codes [diagnosis codes, procedure codes, and medication] and their corresponding medical cost and service detail information [date, type, location, and provider]).

For the hospitalization prediction task, we tested 5 different inputs to a logistic regression model, including (1) demographic and utilization variables; (2) CCS, which groups related medical codes; (3) CCS plus demographic and utilization variables; (4) deep learning, which learns patient representation automatically; and (5) deep learning plus demographic and utilization variables. In addition, we compared the performance of the models with Verscend’s proprietary DxCG Intelligence predictive model (Cotiviti Inc; Atlanta, Georgia), which is considered to be the gold standard in risk stratification for the Medicare population.

For the deep learning model, we utilized the medical codes that had already been semantically embedded into a lower-dimensional space in the previous step using Skip-Gram. This process was achieved without the expert input of any domain knowledge. In contrast, the original DxCG model based on DCGs was developed by the Health Care Financing Administration, now known as CMS, to predict costs for Medicare beneficiaries. Reliant on practitioner expertise and domain knowledge to handcraft machine learning algorithms from raw data, many commercial vendors and ACOs, including PFK, use the DxCG model as the basis for their risk stratification model. At the same time, CCS is a frequently used grouping method for academic research projects as it is not proprietary.

Area under the curve (AUC) was used as the primary evaluation metric. The AUC quantifies the overall ability of the model to discriminate between those individuals who will be hospitalized in the subsequent year and those who will not. It is a balanced consideration of both sensitivity and specificity of the model. An AUC of 100% indicates that the model can correctly categorize the patients 100% of the time (ie, zero false positives and zero false negatives) whereas an AUC of 50% indicates that the model is performing at chance level.

For the all-inclusive models (with all clinical, demographic, and utilization features), we further evaluated their measures of sensitivity, specificity, positive predictive value, and negative predictive value by selecting a range of percentile of high-risk members for next-year hospitalization.

Given the considerable cost associated with hospitalization, we evaluated the financial impact of our model by comparing the actual cost of the patients predicted to be the top 1% (ie, 99th percentile) or 5% (ie, 95th percentile) at risk for hospitalization. In other words, we used hospitalization risk stratification as a proxy for high cost prediction. Using models 1 through 6, we used the data from 2014 and 2015 to predict the risk of hospitalization in 2016. We then selected the top 1% and 5% of the individuals who had the highest predicted risk for hospitalization in 2016 to calculate the actual cost.

From a total study population of 303,404 members, we selected those who had full eligibility with PFK from 2014 to 2016 and were aged between 2 and 18 years during that time. The final data set consisted of 112,641 individuals who were balanced in gender distribution and had a mean age of 8 years. 

 summarizes the demographic and utilization-related characteristics of the study population.

Healthcare utilization in terms of ambulatory and emergency department (ED) visits and hospitalizations remained fairly steady over the 3-year period, whereas the average yearly healthcare expenditure per person steadily increased over the years. See Table 1 for a breakdown of utilization for the study population across years.

Performance of the models generated in this study is summarized in 

. Our result showed that a model using demographic (age, age group, sex) and utilization (previous annual medical cost; number of previous-year hospital, ED, and outpatient visits) features can achieve a relatively good prediction for next-year hospitalization (AUC, 73.1%) (Table 2 [A]).

Deep learning by itself (model 4) outperforms CCS (model 2) and DxCG Intelligence (model 6) on the hospitalization prediction task. When deep learning was coupled with demographic and utilization features (model 5), the best performance was observed (AUC, 75.1%).

Deep learning provides small but meaningful improvements in other measures of predictive performance in addition to the AUC, such as sensitivity and positive predictive value, and comparable performance in specificity and negative predictive value (Table 2 [B]). By selecting the top 10% of high-risk members, the deep learning model had a sensitivity of 0.452 compared with 0.392 to 0.416 for the other models. This means that the model correctly predicted 45.2% of all those who were hospitalized. It is notable that a previous study evaluating the performance of predictive models for a high-cost pediatric population achieved a sensitivity measure of 0.339 with the best model, which included extensive sociodemographic and utilization features along with survey data.

 The models had comparable performance in terms of specificity, which provides a measure of correctly predicting the nonhospitalized members. The positive predictive value is another useful measure when considered in terms of selecting members for care coordination. It reflects the proportion of members predicted to have a high probability of hospitalization (and hence potential targets for preemptive care management) who are actually hospitalized. The deep learning model outperformed the other models in this measure (0.196 vs 0.135-0.166, selecting the top 1%). In contrast, deep learning did not improve the negative predictive value.

Financial Impact of Risk Stratification Using the Hospitalization Prediction Model

In addition to binary prediction of hospitalization or no hospitalization, the logistic regression models described here can also generate risk scores to stratify the patients. As shown in 

, the top 1% of individuals (1126 of the 112,641 patients in the study sample) selected by model 5 (deep learning with demographic and utilization features) had a combined healthcare cost of approximately $31.1 million compared with $26.0 million for those selected by DxCG Intelligence. This finding implies that although model 5 only slightly improves the AUC (Table 2 [A]), the financial impact of its use can be significant. In this case, by using model 5 to select the top 1% of high-risk individuals for care management, the ACO could potentially influence an additional $5 million in costs compared with DxCG Intelligence and $6 million compared with model 1.

This analysis found that the deep learning model plus demographic and utilization characteristics from administrative data was a slightly better risk prediction model than the traditional DxCG model. Deep learning eliminates the reliance on practitioner expertise and domain knowledge to handcraft related medical codes into groups. Depending on the size of the data set, it only takes hours for deep learning to run, in contrast to years of development and refinement of DxCG or CCS by domain experts. This machine learning approach is highly efficient, requiring fewer resources to perform equivalently to or slightly better than current models in a much shorter time. The high-risk patients identified by the deep learning plus demographic and utilization model also incurred much higher annual cost. These findings highlight the potential of deep learning in healthcare data to transform the field of population health management for managed care organizations.

The method and approach developed in this study, although preliminary, have the potential to improve the ability of pediatric ACOs to perform preemptive planning of financial risk. The type of deep learning modeling employed in this study may allow ACOs and others that manage patient populations to more appropriately allocate resources for planning and to reimburse their providers based on enhanced predictability of risk. The ability to improve future diagnostic prediction of a panel of patients may also lead to better assessment of provider performance by taking case mix into consideration.

Most importantly and urgently for patients and population health management, more accurately predicting future risks may lead to more realistic and rationalized planning with resources to meet patient demand. By identifying high-risk patients, physicians and other providers can develop future-oriented preemptive care planning and provide preventive treatment interventions before more serious conditions and events take place. Although we note that prediction is an important but narrow methodological tool, there is the potential for care managers to use other data, such as patient surveys or administrative data that include patients’ socioeconomic and environmental variables, to identify cohorts of patients within the high-risk group identified by the deep learning algorithm who also have other risk factors that can be targeted for intervention, such as a group of children at high risk for increased utilization or costs whose parents screened positive for transportation barriers or who speak a second language at home. Providers may also benefit from risk prediction of their patient population by more proactively planning for cost containment and ensuring that optimized clinical management guidelines are in place.

The present study benchmarked a new predictive learning methodology on a small pediatric ACO data set. The conclusions we can make from this pilot work are limited by the small data set. In addition, we choose to enforce continuous eligibility across all 3 years for patients to be included in this preliminary study. This stringent inclusion criterion may distort the underlying ACO population. Future explorations of the deep learning methods over larger data sets can further validate the findings here. Larger data sets would also allow the inclusion of data from other data sources, such as patient surveys, that include the sociodemographic and environmental variables that affect risk.

While the current analysis focuses on the population level, at the level of the individual patient there is potential to use this deep learning method to model individual patient risks that could be used to guide conversations between patients and providers. The risks to be modeled can also be extended from hospitalization to readmission or claim frauds. There is a caveat to be considered when extending this work: The deep learning approach limits the interpretability of the model, as it is hard to decipher how the variables contribute to the deep learning algorithm used to classify high-risk groups.

An additional limitation is that the risk prediction models used to benchmark this new methodology were limited to those available to the study team. We acknowledge that other potential case mix algorithms, such as the Johns Hopkins ACG, have been applied and evaluated in Medicaid populations. However, the ACG is proprietary, whereas the CCS was freely available and the DxCG was the commercial model used by PFK at the time of this study. It is also important to note that our cost savings predictions may be an upper-bound estimate because implementing new care management approaches based on risk identification via the deep learning model would incur additional costs to the system.

From the perspective of population health management, the deep learning approach to risk stratification has clear implications that include (1) enabling the discussion of future policies of risk adjustment based on population risk stratification; (2) providing a generalizable and scalable platform to identify trends in disease occurrence over time, ensuring timely public health investigation and intervention; and (3) improving healthcare utilization forecasting, thus enabling an ACO or other care management entity to identify gaps in healthcare resource allocation for its patient population.

Although deep learning analytic approaches are steadily increasing, the application of these models to risk stratification has been slow to gain traction in healthcare delivery systems. Fortunately, some forward-looking accountable care systems continue to support and conduct research that demonstrates the “value-add” of deep learning methods that may enhance predictive modeling capability and thus ensure better treatment and financial outcomes. In order to achieve more rapid adoption of these innovative methods, several users and perspectives will have to be considered. We have conducted an initial study of this type of modeling versus more traditional methods and found very favorable initial results.

The authors would like to thank Brad Stamm from Partners for Kids for valuable discussion on model evaluation. They also thank Drs Stephen Cardamone, Kelly Kelleher, Deena Chisolm, Ann McAlearney, and Sven Bambach for providing thoughtful feedback on the manuscript.

 Research Information Solutions and Innovation, The Research Institute at Nationwide Children’s Hospital (EJDL, XZ, SM, SML), Columbus, OH; Health Services Management and Policy, College of Public Health (JLH, TH), and Department of Biomedical Informatics and Department of Pediatrics, College of Medicine (SML), The Ohio State University, Columbus, OH; Department of Electrical Engineering and Computer Science, Ohio University (XZ, CL), Athens, OH; Partners for Kids (JK), Columbus, OH.

Artificial intelligence based on medical claims data outperforms traditional models in stratifying patient risk.

Jennifer L. Hefner, PhD, MPH

Articles

Approaches for Overcoming Barriers to Cross-Sector Data Sharing

A Deep Learning Model for Pediatric Patient Risk Stratification