Hector P. Rodriguez, PhD, MPH

 An increasing number of federal initiatives allow states flexibility in selecting the strategies used to achieve initiative-specific goals.&ensp;Variation in the foci and intensity of implementation may explain why federal policy initiatives succeed in some states and fail in others.  The CMS State Innovation Models (SIM) initiative is a complex policy intervention implemented with substantial variation across states and may have variable impacts. This paper presents a method to characterize and account for that variation in states’ implementation foci and intensity in natural policy experiments.

 A combination of quantitative and qualitative measures of SIM implementation was used to characterize the foci of payment and delivery system reforms across states.

 A modified Delphi expert panel process was used to prioritize the features of SIM implementation that would differentiate grantee states with respect to improved health outcomes. Three researchers then reviewed summaries of published evaluations and reports to characterize and score states on each implementation feature. Expert panelists guided the researchers on developing the criteria and weights applied to the focus areas when calculating SIM implementation intensity scores for states.

 Over 3 years of an expert panel process, 4 dimensions of SIM implementation that would most affect health outcomes were prioritized: (1) extent and breadth of stakeholder engagement, (2) extent that SIM implementation was focused on improving behavioral health, (3) amount of SIM funding per capita, and (4) breadth and depth of value-based payment reforms. Scoring states based on the prioritized factors resulted in composite scores that differentiated states into 3 categories: high, moderate, and low implementation intensity.

 We developed a stakeholder-driven method to measure and account for variation in implementation foci and intensity in a federal policy initiative that was implemented heterogeneously across grantee states. Our method for characterizing state implementation variation may be useful for natural policy experiments examining the variable impact of policy initiatives.

The American Journal of Accountable Care. 2019;7(3):12-17

Over the past decade, health policies and programs intended to spur innovation in delivery system design and payment reform have become commonplace across the United States.

 Studies examining the effect of state health policies rely on natural experiment study designs, but they do not account for differences in states’ foci and experiences of policy implementation. Characterizing states as exposed or not exposed (1 or 0), as is traditionally done in natural experiments of state health policy initiatives, is overly simplistic and does not consider the specific strategies used by states. Ideally, features of each state’s rollout, including reform foci and intensity of activities, could be modeled quantitatively. The small number of states involved in any given reform, however, precludes the use of quantitative methods to produce a taxonomy to characterize “types” of policy implementation using k-means cluster analysis or another data reduction method.

 As part of a natural experiment of the federal—state program—the CMS State Innovation Models (SIM) initiative&mdash;we describe a stakeholder-driven method to prioritize, assess, and account for state-level variation in natural policy experiments.

The SIM initiative awarded funding and technical assistance to states through a competitive process. State health departments proposed plans to implement innovative delivery and payment models to improve health system performance, improve the quality of patient care, and decrease healthcare costs for all residents of the state. Through SIM, the federal government provided states with more than $1 billion in funding and substantial technical assistance to plan, pilot test, and implement payment and delivery system reforms.

 Round 1 of SIM funding was awarded in April 2013 to 6 states (Arkansas, Maine, Massachusetts, Minnesota, Oregon, and Vermont). Round 2 was awarded in December 2014 to 11 additional states (Colorado, Connecticut, Delaware, Idaho, Iowa, Michigan, New York, Ohio, Rhode Island, Tennessee, and Washington). Some states that applied for, but did not receive, SIM funding were awarded modest planning grants ($3 million or less) to aid in advancing their innovations to the potential testing phase in the future. This staged roll-out of SIM allows for a natural experiment study design to evaluate the impact of this policy on population health outcomes.

Previous reviews have conceptualized the critical role of variation in implementation processes to understand differential impacts of policy change.

 Implementation science considers intensity and other aspects of the implementation process, including the adoption, reach, and fidelity of implementation to intended policy features.

 The application of implementation science in health services and policy research is growing, but it primarily focuses on the ways in which practitioners successfully incorporate new policies into routine practice as study outcomes.

 Studies have rarely examined how federal policies are differentially implemented at the state level and how these variations affect healthcare utilization and health outcomes.

The political science subfield of policy implementation research analyzes sources of variation in the implementation of large-scale policies (ie, laws and regulations) and does consider policy goals such as health outcomes as dependent variables, but as with the other perspectives, it does not study how the variation itself influences these outcomes. A handful of policy implementation research studies have described variation in the focus of state-level policy implementation, including applications to welfare policies, medical marijuana policies, and youth sports traumatic brain injury policies.

 However, we could find no empirical studies that simultaneously characterized the foci and intensity of state-level policy implementation—considerations that are critically important for understanding the impacts of a complex, multifaceted policy intervention like SIM.

Our conceptualization of the connection between policy implementation and outcomes is most similar to that of Strehlenert and colleagues’ Conceptual Model for Evidence-Informed Policy Formulation and Implementation,

 which covers the entire policy process from agenda setting and policy formulation to implementation and outcomes evaluation; however, this framework was used only descriptively with case studies and not to make comparisons across multiple implementers. CMS allowed states considerable latitude in SIM plan foci and implementation strategies,

 and this variation in policy implementation could result in differential impacts of SIM on utilization and health outcomes across the grantee states. To advance the examination of heterogeneous effects in natural policy experiments, we developed a stakeholder-driven method to measure and account for variation in implementation foci and intensity in a federal policy initiative that was implemented heterogeneously across states.

We used a combination of quantitative and qualitative measures to prioritize, classify, and analyze SIM implementation variation for each of the 17 grantee states. To do this, we convened an expert advisory panel composed of 8 SIM leaders from different states to provide us with qualitative and quantitative input about core SIM activities and, ultimately, to participate in a modified Delphi expert panel process to prioritize key differences in implementation foci and strategies across the SIM states. The panel members were recruited from 8 SIM grantee states: Arkansas, Colorado, Iowa, Oregon, Maine, Michigan, Minnesota, and Washington.

 The research team facilitated web-based quarterly meetings from October 2015 to January 2019. Webinars were recorded and transcribed. Several important policy implementation differences were identified and discussed during the first 3 meetings. Importantly, states varied in the delivery system and payment reforms that were tested (

 summarizes examples). Some states, such as Minnesota and Colorado, emphasized delivery system reform, including using SIM initiative funding to implement patient-centered medical homes, integration of physical and behavioral healthcare, use of health information technology, and/or health information exchange; meanwhile, Washington emphasized implementing value-based payment reforms, such as shared savings and total cost of care models.

 Another important factor discussed was that states were allowed substantial latitude with regard to the distribution of SIM funds within the state. For example, some states, such as Arkansas, retained all of the funds at the state level to support and augment pre-existing programs in physician practice transformation and Medicaid innovation. Other states, such as Minnesota, distributed most of the funding down to the local and regional levels through competitive grants. Maine used a competitive process to contract with several statewide organizations to pursue statewide health system transformation efforts.

Panelists emphasized that the role of SIM in each state additionally differed based on states’ prior investments in healthcare delivery and payment reform. In some states, the resources were used to establish new health system infrastructure. For example, Washington implemented regionally organized public/private Accountable Communities for Health and created a new “support hub” for practice transformation. In other states, SIM resources were used primarily to accelerate changes that were already under way in the state and used the funding to improve interagency alignment and coordination. For example, Maine created a governance structure for the 6 strategic pillars that it selected, convened decision makers from across the state to take action on the proposed innovations, and used this governance structure to ensure that implementing the SIM initiative was a priority of Maine’s Department of Health and Human Services.

Another important difference across states discussed by panelists was changes in requirements communicated by CMS for round 2 grantees. In round 1, states had high latitude in selecting their performance indicators and targets, as long as they made a strong case as to why their foci of activities would improve these indicators. By round 2, however, CMS was more prescriptive in the performance indicators and made tobacco use, obesity, and diabetes required indicators. In addition, to ensure coordination and linkages with overall state policies, round 2 applications were required to be routed through the state’s governor’s office for approval prior to submission. The absolute amount awarded to each state was greater, on average, in round 2 ($56.6 million) compared with round 1 ($42.4 million).

Based on observations and data from the first 3 meetings, a list of the 10 most important factors that panelists agreed most differentiated states with respect to SIM implementation was finalized: (1) amount of SIM funding received by state, per capita; (2) whether the state was funded in SIM round 1 or round 2; (3) extent to which SIM implementation was focused on improving behavioral health; (4) extent to which SIM implementation was focused on diabetes; (5) breadth and depth of value-based payment reforms; (6) extent to which SIM funds were centralized versus distributed to local/regional entities; (7) co-occurring delivery system interventions, such as the Section 1115 Medicaid demonstration programs; (8) co-occurring Medicaid expansion; (9) co-occurring philanthropic contributions; and (10) state agency funding reallocation.

Then, 7 of the panel members engaged in a 3-round modified Delphi expert panel prioritization process,

 which involved ranking the 10 factors using Qualtrics survey software (Qualtrics; Provo, Utah) based on their relevance for differentiating impactful versus unsuccessful implementation of the SIM initiative among grantee states. The first round of survey results was discussed during a subsequent meeting and used as a basis for modifying/adding/dropping factors for the second round of the ranking process, with the aim of achieving convergence. Panelists were allowed to add additional criteria in the first and second rounds, resulting in the inclusion of stakeholder engagement as an important dimension of implementation after round 1 of the expert panel process. After 3 rounds of ranking, criteria were developed in consultation with panelists to characterize the implementation intensity and resources for the top 4 prioritized factors for each of the 17 states. Finally, we assessed each state’s efforts with regard to these factors using information from state agency reports of SIM implementation, RTI International national evaluation reports,

 CMS, and the US Census Bureau. This study was approved by the University of California, Berkeley, Committee for the Protection of Human Subjects.

The modified Delphi process was completed in 3 rounds (

) and resulted in the prioritization of 4 implementation factors, in order of importance: (1) extent and breadth of stakeholder engagement; (2) extent to which SIM implementation was focused on improving behavioral health; (3) amount of SIM funding received by state, per capita; and (4) breadth and depth of value-based payment reforms.

Once prioritized, intensity levels for each implementation factor were determined in consultation with the panelists. In terms of stakeholder engagement, panelists indicated that interagency coordination and working well with community-based organizations were central to getting broad-based delivery system and payment reforms launched and implemented broadly. After extensive discussion, panelists concluded that the document review and interview methods would be inadequate for assessing stakeholder engagement given the complex web of organizations and agencies involved in implementing SIM. Because stakeholder engagement is a contextual influence on policy implementation rather than about implementation foci or resources, panelists recommended that it should not be factored into the calculation of the SIM implementation intensity index.

 Data on SIM funding per capita were obtained from CMS and the US Census Bureau, and each state’s level of per capita funding was assigned a numeric value according to whether it fell within the lowest (1), middle (2), or highest (3) third of the distribution.

To measure the extent of focus on behavioral health and the breadth and depth of payment reforms, panelists confirmed the use of published evaluations as the best sources for characterizing these activities. Accordingly, we created summaries of each state’s efforts based on review of the comprehensive evaluation reports by RTI International,

 as well as states’ publications on their plans and progress. Using these summaries, 3 evaluators on the research team independently rated each state’s efforts in these domains according to predefined rubrics. The extent of behavioral health focus was rated 1 for little to no focus, 2 for some focus, or 3 for strong focus. The breadth and depth of payment reform was rated 1 if payment reform was not part of the state’s SIM plan, 2 if 1 payer participated, or 3 if more than 1 payer participated (including at least Medicare or Medicaid).

Panelists provided feedback about how the weight of each factor should contribute to an overall index of SIM implementation intensity. Behavioral health focus and depth of payment reform were deemed by panelists to be more important for outcomes than per capita funding because the funding level per capita is quite low; the grant simply provided foundational resources, and states had to have the wherewithal to leverage these resources. As a result, these categories were assigned a weight of 40% and per capita funding was assigned a lower weight of 20%. These final weighted summary scores exhibited low variation; the vast majority of states had scores that fell between 1.8 and 2.2 (

). However, negative outliers were identified as those that scored below 1.8 (Connecticut, Michigan, and Iowa) and positive outliers as those that scored above 2.2 (Delaware, Maine, and Colorado). Due to the concentrated distribution, SIM states were grouped into 3 categories based on their behavioral health focus, breadth and depth of payment reform efforts, and per capita funding. The resulting scoring and categorization was shared with panelists for their review and feedback, resulting in requested changes to reclassify a state’s payment reform activities and to modify the weighting criteria. When examined, the 2 changes did not affect the categorization of SIM states (Table 3).

There is a trend toward more state-level health policy development and implementation in the United States, with less emphasis on homogenous federal reforms. Given the inherent challenges posed by the naturally occurring variation in such natural policy experiments, new methods are needed to take into account differences in policy implementation. We describe a method based on Delphi assessments and analysis of source documents relevant to policy implementation that may be useful for assessing differential impacts of SIM across states. Importantly, the SIM implementation factors prioritized by the expert panel process included the extent of behavioral health integration and the breadth and depth of value-based payments, which were recently found to be important differentiators of SIM implementation in national evaluations of round 1 SIM states.

 As our natural experiment research moves forward, we will directly examine the impact of implementation variation on healthcare utilization and outcomes. We hypothesize that SIM states with high implementation intensity for the prioritized areas and greater resources will achieve relatively greater reductions in preventable utilization and improved patient outcomes.

Federal policy initiatives that allow states flexibility in their foci and implementation strategies can have heterogeneous impacts on health system performance and patient outcomes. We developed a stakeholder-driven method to measure and account for variation in implementation foci and intensity in a federal policy initiative that was implemented heterogeneously across grantee states. Our method for characterizing state implementation variation may be useful for natural policy experiments examining the variable impact of policy initiatives across states and can be used alongside other important state-level factors, such as socioeconomic profiles and political contexts. We encourage a dialogue among policy makers, implementers, and evaluators of state health policy reforms to unpack the role of implementation variation in explaining outcomes of broad-based policy changes.

The authors are grateful for critical feedback from their project’s advisory panel of SIM directors and designees. They also thank Stephen M. Shortell, PhD, MPH, MBA, for critical input on the modified Delphi expert panel process and Amalya Penso for assistance with fielding and analyzing the survey.

 Department of Family and Community Medicine, Philip R. Lee Institute for Health Policy Studies, University of California, San Francisco (DRR), San Francisco, CA; Center for Healthcare Organizational and Innovation Research, School of Public Health, University of California, Berkeley (AZP, SB, HPR), Berkeley, CA.

 This publication was made possible by the NEXT-D2 (Natural Experiments for Translation in Diabetes 2.0) Study, a cooperative agreement (5U18DP006123) jointly funded by the CDC and the National Institute of Diabetes and Digestive and Kidney Diseases.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (HPR); acquisition of data (DRR, HPR); analysis and interpretation of data (DRR, AZP, SB, HPR); drafting of the manuscript (DRR, AZP, HPR); critical revision of the manuscript for important intellectual content (DRR, AZP, SB, HPR); obtaining funding (HPR); administrative, technical, or logistic support (SB); and supervision (SB, HPR).

 Hector P. Rodriguez, PhD, MPH, Center for Healthcare Organizational and Innovation Research, School of Public Health, University of California, Berkeley, 2121 Berkeley Way, Room 5302, Berkeley, CA 94720-7360. Email: hrod@berkeley.edu.

1. Shortell SM, Rittenhouse D. The most critical health care issues for the next president to address. 

. 2016;165(11):816-817. doi: 10.7326/M16-2471.

2. Wu FM, Shortell SM, Lewis VA, Colla CH, Fisher ES. Assessing differences between early and later adopters of accountable care organizations using taxonomic analysis. 

. 2016;51(6):2318-2329. doi: 10.1111/1475-6773.12473.

3. Hughes LS, Peltz A, Conway PH. State Innovation Model initiative: a state-led approach to accelerating health care system transformation. 

. 2015;313(13):1317-1318. doi: 10.1001/jama.2015.2017.

4. Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. 

. 2014;12:34. doi: 10.1186/1478-4505-12-34.

5. Strehlenert H, Richter-Sundberg L, Nyström ME, Hasson H. Evidence-informed policy formulation and implementation: a comparative case study of two national policies for improving health and social care in Sweden. 

. 2015;10:169. doi: 10.1186/s13012-015-0359-1.

6. Slaughter SE, Hill JN, Snelgrove-Clarke E. What is the extent and quality of documentation and reporting of fidelity to implementation strategies: a scoping review. 

. 2015;10:129. doi: 10.1186/s13012-015-0320-3.

7. Fisher ES, Shortell SM, Savitz LA. Implementation science: a potential catalyst for delivery system reform. 

. 2016;315(4):339-340. doi: 10.1001/jama.2015.17949.

8. Nilsen P, Ståhl C, Roback K, Cairney P. Never the twain shall meet?—a comparison of implementation science and policy implementation research. 

. 2013;8:63. doi: 10.1186/1748-5908-8-63.

9. Chapman SA, Spetz J, Lin J, Chan K, Schmidt LA. Capturing heterogeneity in medical marijuana policies: a taxonomy of regulatory regimes across the United States. 

. 2016;51(9):1174-1184. doi: 10.3109/10826084.2016.1160932.

10. De Jong GF, Graefe DR, Irving SK, St Pierre T. Measuring state TANF policy variations and change after reform. 

. 2006;87(4):755-781. doi: 10.1111/j.1540-6237.2006.00432.x.

11. Coxe K, Hamilton K, Harvey HH, Xiang J, Ramirez MR, Yang J. Consistency and variation in school-level youth sports traumatic brain injury policy content. 

. 2018;62(3):255-264. doi: 10.1016/j.jadohealth.2017.07.003.

12. McKernan SM, Bernstein J, Fender L. Taming the beast: categorizing state welfare policies: a typology of welfare policies affecting recipient job entry. 

. 2005;24(2):443-460. doi: 10.1002/pam.20102.

13. Kissam SM, Beil H, Cousart C, Greenwald LM, Lloyd JT. States encouraging value-based payment: lessons from CMS’s State Innovation Models initiative. 

. 2019;97(2):506-542. doi: 10.1111/1468-0009.12380.

14. Beil H, Feinberg RK, Patel SV, Romaire MA. Behavioral health integration with primary care: implementation experience and impacts from the State Innovation Model round 1 states. 

. 2019;97(2):543-582. doi: 10.1111/1468-0009.12379.

15. The Lewin Group, Inc. Maine State Innovation Model self evaluation: year three final report. Maine website. maine.gov/dhhs/sim/evaluation/documents/ME%20SIM%20Self%20Evaluation%202016%20Final%20Report%2012.21.16.pdf. Published December 21, 2016. Accessed August 2, 2019.

16. Kennedy HP. Enhancing Delphi research: methods and results. 

. 2004;45(5):504-511. doi: 10.1046/j.1365-

17. RTI International. State Innovation Models (SIM) initiative evaluation: model test year three annual report. CMS website. downloads.cms.gov/files/cmmi/sim-rd1mt-thirdannrpt.pdf. Published September 2017. Accessed August 2, 2019.

Articles

This paper presents a method to characterize policy implementation across states to enable more nuanced impact assessments of federal healthcare delivery system and payment reforms.

Implementation Variation in Natural Experiments of State Health Policy Initiatives

 The adoption of advanced health information technology (HIT) capabilities, such as predictive analytic functions and patient access to records, remains variable among healthcare systems across the United States. This study is the first to identify characteristics that may drive this variability among health systems.

Responses from the 2017/2018 National Survey of Healthcare Organizations and Systems were used to assess the extent to which healthcare system organizational structure, electronic health record (EHR) standardization, and resource allocation practices were associated with use of 5 advanced HIT capabilities. Of 732 systems surveyed, 446 responded (60.9%), 425 (58.1%) met sample inclusion criteria, and 389 (53.1%) reported consistent EHR use.

 Measures of adoption, resource allocation, and organizational structure were developed based on survey responses. Multivariate linear regression with control variables estimated the relationships.

 Adoption of advanced HIT capabilities is low and variable, with a mean of 2.4 capabilities adopted and only 8.4% of systems reporting widespread adoption of all 5 capabilities. In adjusted analyses, EHR standardization (β = 0.76; 

 = .001) was the strongest predictor of the number of advanced capabilities adopted, and ownership and management of medical groups (β = 0.32; 

 = .04) was also a significant predictor.

 Health systems that standardize their EHRs and that own and manage hospitals and medical groups have higher rates of advanced HIT adoption and use. System leaders looking to increase the use of advanced HIT capabilities should consider ways to better standardize their EHRs across organizations.

The adoption of advanced health information technology (HIT) capabilities, such as predictive analytic functions and patient access to records, varies among healthcare systems across the United States. This study is the first to identify characteristics that may drive this variability among health systems.

Multivariate regression analysis was used to determine the extent to which organizational structure, resource allocation practices, and electronic health record (EHR) standardization predicted HIT adoption.

EHR standardization was the strongest predictor of advanced HIT adoption. Owning and managing medical groups and hospitals was also a statistically significant predictor of advanced HIT adoption.

Health system leaders looking to improve the rates of advanced technology adoption may consider ways to standardize their EHR systems across organizations.

Since the passage of the Patient Protection and Affordable Care Act in 2010, health systems, hospitals, and medical groups have faced considerable public pressure

 to assume an increased amount of risk for the costs and outcomes of their patient populations through value-based payment models. This increased risk requires transforming the way that healthcare services are delivered and reimagining the role that health systems play in managing their patient populations. One response has been the development of more than 1000 accountable care organizations (ACOs) in the 50 states, providing care to nearly 33 million Americans.

 A key capability of such organizations is their use of advanced health information technology (HIT),

 but adoption of HIT capabilities necessary to enable value-based care remains slow.

 Although vendors have developed a wider variety of additional capabilities to facilitate patient engagement and performance measurement, wide variability in the uptake of these features by healthcare organizations has caused increasing concerns about a “digital advanced use divide.”

Given the extensive and well-documented challenges that come with driving innovations in health systems,

 it seems likely that at least some of this variation can be explained by organizational characteristics. Previous studies in medical groups, for example, have found hospital ownership to be predictive of basic HIT adoption and care management processes in medical groups,

 and others have found that larger, more integrated ACOs are better positioned for technology adoption.

 These findings generate further questions regarding the heterogeneity of HIT implementation among health systems, such as how they allocate resources for HIT development and the extent to which they standardize their use of HIT.

There is considerable heterogeneity in organizational models used by health systems. In this study, health systems were defined as corporate parents or owner subsidiaries that include either 1 hospital and at least 1 set of primary care physicians, 2 or more hospitals, or 2 or more primary care groups and are connected by contract or ownership. Systems may own and manage hospitals and medical practices associated with the system or may operate under affiliate models. It is possible that health systems with more direct management responsibilities of their hospitals and medical groups may more effectively integrate different stakeholders, a critical element of innovation in healthcare organizations,

 and may be able to deliver more training across the enterprise, another critical success factor of electronic health record (EHR) implementations identified in previous research.

Second, resource allocation practices can vary widely across health systems. Health systems that centrally allocate resources and purchase software may realize economies of scale with their purchases and have more bargaining power with vendors, which would enable greater adoption of advanced capabilities. In addition, they may have greater resources to invest in enterprise-wide training. Existing research provides some empirical evidence of the impact of resources on HIT adoption in health systems,

 and there is also evidence that information technology (IT) resources have driven adoption and success of other types of software systems in large enterprises.

Finally, there may be differences in the extent to which health systems standardize their use of technology. Health systems that standardize their EHRs will likely need to change fewer processes and deliver fewer different versions of training when rolling out new capabilities, which may enable greater adoption. Although the link between enterprise-wide technology standardization and adoption has not been empirically examined in health systems, researchers have documented the importance of enterprise-wide architecture and standards for other IT capabilities within large organizations.

Drawing on the conceptual model shown in the 

, the following 3 hypotheses were tested: (1) Compared with health systems that do not own and manage medical groups and hospitals, health systems that own and manage medical groups and hospitals will have greater adoption of advanced HIT capabilities; (2) Compared with health systems with decentralized resource allocation practices, health systems with more centralized resource allocation practices will have greater adoption of advanced HIT capabilities; and (3) Compared with health systems with low EHR standardization, health systems with higher degrees of EHR standardization will have greater adoption of advanced HIT capabilities.

Data used were from the system version of the 2017/2018 National Survey of Healthcare Organizations and Systems (NSHOS), a nationally representative sample of healthcare systems that took place from June 2017 to August 2018. The survey was based on previous surveys developed involving the National Study of Physician Organizations

 along with additional questions focusing on the use of biomedical innovations, patient engagement strategies, evidence-based care management, and performance-based incentives. Respondents were the individuals most knowledgeable about the survey questions, who most commonly were system presidents, chief executive officers, or chief medical officers. Survey responses were matched with data obtained from IQVIA information services to provide additional context on systems’ organizational characteristics. Of the 732 randomly sampled healthcare systems, 446 responded (60.9% completion), 425 of which were included after data cleaning was completed. Given the strong financial incentives that the Meaningful Use program created for basic EHR adoption, we eliminated systems that had not fully adopted basic EHRs (n = 36). The final analytic sample included 389 healthcare systems, 22.6% (n = 88) of which were subsidiaries of corporate parents.

 Survey respondents were asked about the adoption and use of 5 advanced HIT capabilities across their health systems. Four of these capabilities—patients’ access to their electronic medical records, patients’ ability to electronically comment on their medical records, physicians’ and patients’ ability to communicate with each other via secure email, and physicians’ ability to know whether patients have filled prescriptions&mdash;are required capabilities for Certified Electronic Health Record Technology (CEHRT) designation.

 The fifth, advanced analytic systems, has been identified in previous research as a key ingredient to successful care management for complex patients.

For each of these capabilities, respondents were asked how many hospitals or medical groups in their systems had the following features, with possible answers being “none,” “some,” “most,” or “all.” To construct a measure of successful advanced HIT adoption, we assigned 1 point to answers of “most” or “all” and 0 points to answers of “none” or “some.” Overall adoption was then measured on a scale of 0 to 5, with a score of 0 indicating no capabilities had been adopted successfully and a score of 5 indicating all capabilities were adopted successfully.

 Health system ownership and management was measured via 2 survey questions that asked survey respondents about whether the system owned or managed hospitals and medical groups. Respondents had the option of answering “no,” “own only,” “manage only,” and “own and manage.” Because we were sampling health systems, which, by nature, had a high level of medical group and hospital ownership and management (nearly 50% of the systems in our sample reported both owning and managing hospitals and medical groups), a categorical variable was created, assigning a value of 1 to systems that owned and managed hospitals and medical groups and 0 to all other systems.

The degree to which a health system centrally allocated resources was measured using answers to 3 questions in the survey based on their ability to serve as potential proxies for resources that could be dedicated to new HIT systems. The questions were “What level of financial planning and revenue sharing best describes your healthcare system?” “At what level [is capital budgeting] primarily conducted?” and “At what level [is IT vendor selection] primarily conducted?” We assigned a point value of 1 for each activity conducted at the system level (and 0 for those conducted locally or regionally) and took an unweighted average of the point values across the 3 questions to develop an index of how much resource allocation was centralized (0 being none of these activities conducted at the system level, 1 being all activities conducted at the system level).

EHR standardization was measured using answers to 2 questions: “How many EHR systems do you have in place across your hospitals’ primary care groups?” and “To what extent are EHR data elements standardized?” Fully standardized EHR data elements and a single EHR system were each assigned a point value of 1, and we took an unweighted average of the point values across the 2 questions to develop an index of how much their EHR systems were standardized.

 Previous research has found that alternative payment model (APM) participation is correlated with both more integrated organizational structures

 We controlled for APM experience, which was measured by the total number of APMs in which most or all of the system’s hospitals and medical groups were participating. We also controlled for relative size (standardized to a mean of 0 and SD of 1), as measured by the number of physicians in the system, and whether respondents perceived there to be high levels of competition in either inpatient or outpatient settings. We also controlled for each system’s geographic region based on the US Census, which was categorized as South, Midwest, Northeast, West, or Multiregion based on the states in which they operated.

Descriptive analyses of all key study variables were conducted, including a breakdown of advanced HIT adoption, organizational structure, EHR standardization, and resource allocation practices by number and percentage of systems. All 3 hypotheses were tested simultaneously using multivariate regression models that examined the association of organizational structure, resource centralization, and EHR standardization with advanced HIT adoption, controlling for system size, perceived competition, region, corporate parent status, and APM experience. Results were weighted to account for differential selection probabilities resulting from probability sampling of systems based on organizational complexity, including having a subsidiary or not.

 provides descriptive statistics for key study variables. Adoption of advanced HIT capabilities was inconsistent, with a mean of 2.4 capabilities adopted and most (80.5%) systems in our sample adopting between 1 and 4 features. Only 8.4% of systems in our sample reported adopting all 5 advanced HIT capabilities. Adoption rates varied widely by feature, with more than 75% of systems reporting the ability of patients to access their medical records but less than 32% reporting the ability of physicians to know when patients fill prescriptions and of patients to comment on their medical records.

Just under half (48.8%) of the systems in our sample owned and managed their hospitals and medical groups. They also allocated resources centrally; 67.3% reported conducting capital budgeting at the system level and 78.9% reported conducting vendor selection at the system level. Most systems used a single EHR (69.0%) and standardized EHR data elements (58.6%), although just under half reported using both a single EHR and standardized data elements (48.1%). APM experience varied, with the lowest rate of participation being in risk-bearing ACOs (21.5%) and the highest rate of participation being in primary care improvement and pay-for-performance programs (60.4%).

Multivariate regression results indicate that the degree of EHR standardization (β = 0.76; 

 = .001) was the strongest predictor of advanced HIT adoption, supporting our third hypothesis, and system management and ownership of hospitals and medical groups was also statistically significant (β = 0.32; 

 = .04), supporting our first hypothesis. The degree of centralized resource allocation was not a significant predictor of advanced HIT adoption (β = 0.42; 

 = .13). Of our control variables, APM experience was a significant predictor of advanced HIT adoption (β = 0.10; 

 = .006), but other variables were not significant. Full regression results can be found in 

Our findings have several implications for the further adoption of advanced HIT capabilities by health systems. First, the strongest predictor of advanced HIT adoption in a health system is the extent to which EHR systems are standardized. The second is that ownership and management of hospitals and medical groups is also a significant predictor of advanced HIT adoption. The third is that system resource allocation practices are less significant predictors of advanced HIT adoption when controlling for other organizational characteristics. Whereas previous studies’ findings suggest that resources and economies of scale are the primary drivers of HIT adoption among medical groups and hospitals, we found that EHR standardization, rather than centralized resource allocation, was the strongest predictor of advanced HIT adoption across a healthcare system.

There are several reasons why standardization may have a much stronger association with advanced HIT adoption than resource centralization in our study. The first is that our focus was specifically on health systems, which, by nature, are more centralized and well-resourced organizations than are independent hospitals and medical groups. The second is that the adoption of advanced EHR capabilities may represent more a challenge of change management than of resource allocation. All of the systems in our analytic sample had already acquired EHRs, and 4 of the 5 capabilities in question were required capabilities of EHR systems for CEHRT designation. Third, health systems may decide to standardize their EHR systems so they can adopt advanced capabilities in the future, which the cross-sectional nature of our study could not address.

Our findings indicate that health systems that aim to accelerate the adoption of advanced HIT capabilities may benefit from standardizing their EHR systems across hospitals and medical groups within the system. The results related to resource allocation indicate that organizations with more distributed forms of resource allocation, but high levels of standardization, may achieve similar levels of advanced technology adoption. The findings also suggest that payers can assist health systems with EHR adoption by targeting technical assistance toward health systems with lower levels of EHR standardization. Moreover, HIT vendors may consider developing ways to standardize use of their products across systems to ensure greater adoption of new and beneficial features.

The results should be considered in light of some limitations. First, the cross-sectional nature of the NSHOS cannot establish the temporal ordering of any associations found. Second, NSHOS is a single-informant survey, which may affect the internal validity of the study. Self-reported data are sometimes inaccurate, and it is possible that the reliability with which systemwide standardization, resource allocation, and HIT capabilities are reported varied by respondent. Third, data limitations prevented us from controlling for additional factors that may influence advanced HIT adoption, such as patient mix. Finally, we included measures of centralized resource allocation, but the specific resources available for investment in training and software upgrades were not assessed. Questions about targeted HIT investments would be useful to include in future research.

The degree of EHR standardization within health systems, as measured by the degree of uniformity of technology systems and data elements across hospitals and medical groups, is a stronger predictor of advanced HIT adoption than the system’s ownership and management structure, resource allocation practices, or APM participation. Health system leaders looking to improve the diffusion of new technologies should consider ways to better standardize their implementation and use of EHRs to drive widespread adoption of and benefit from new features. Further research should assess the impact of healthcare system resources for training and software upgrades on the adoption of advanced HIT and determine the drivers of wide variability in the adoption of individual advanced HIT capabilities.&ensp;

The statements, findings, conclusions, views, and opinions contained and expressed in this article are based in part on data obtained under license from IQVIA information services: OneKey subscription information services 2010-2017, IQVIA Incorporated, all rights reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA Incorporated or any of its affiliated or subsidiary entities.

The American Medical Association (AMA) is the source for the raw physician data; statistics, tables, or tabulations were prepared by Paul Norton using AMA Masterfile data.

 School of Public Health, University of California Berkeley (PTN, HPR, SMS), Berkeley, CA; Gillings School of Global Public Health, University of North Carolina Chapel Hill (VAL), Chapel Hill, NC.

 This study was supported by the Agency for Healthcare Research and Quality Comparative Health System Performance Initiative under grant #1U 19HS024075, which studies how healthcare delivery systems promote evidence-based practices and patient-centered outcomes research in delivering care.

 Concept and design (PTN, HPR, SMS, VAL); acquisition of data (HPR, SMS, VAL); analysis and interpretation of data (PTN, HPR, VAL); drafting of the manuscript (PTN); critical revision of the manuscript for important intellectual content (PTN, HPR, SMS, VAL); statistical analysis (PTN); obtaining funding (HPR, SMS, VAL); administrative, technical, or logistic support (PTN); and supervision (HPR, SMS).

 Paul T. Norton, MPH, MBA, School of Public Health, University of California Berkeley, 2121 Berkeley Way, Berkeley, CA 94704. Email: paul_norton@berkeley.edu.

1. Burwell SM. Setting value-based payment goals—HHS efforts to improve U.S. health care. 

. 2015;372(10):897-899. doi: 10.1056/NEJMp1500445.

2. Muhlestein D, Saunders R, Richards R, McClellan M. Recent progress in the value journey: growth of ACOs and value-based payment models in 2018. 

 Blog website. healthaffairs.org/do/10.1377/hblog20180810.481968/full. Published August 14, 2018. Accessed August 17, 2018.

3. King J, Patel V, Jamoom E, DesRoches C. The role of health IT and delivery system reform in facilitating advanced care delivery. 

4. Adler-Milstein J, Embi PJ, Middleton B, Sarkar IN, Smith J. Crossing the health IT chasm: considerations and policy recommendations to overcome current challenges and enable value-based care. 

. 2017;24(5):1036-1043. doi: 10.1093/jamia/ocx017.

5. Adler-Milstein J, Holmgren AJ, Kralovec P, Worzala C, Searcy T, Patel V. Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divide. 

. 2017;24(6):1142-1148. doi: 10.1093/jamia/ocx080.

6. Berwick DM. Disseminating innovations in health care. 

. 2003;289(15):1969-1975. doi: 10.1001/jama.289.15.1969.

This is the first national study to examine the relationship between healthcare system organizational characteristics and adoption of advanced health information technology capabilities.

Organizational Influences on Healthcare System Adoption and Use of Advanced Health Information Technology Capabilities

 In 2011, the California Right Care Initiative implemented a countywide physician organization learning collaborative called University of Best Practices (UBP) in San Diego County for major healthcare systems and physician organizations to share best practices in managing cardiovascular and cerebrovascular risk factors. Our objective was to examine whether UBP was associated with fewer hospitalizations for heart attacks and strokes.

 A quasi-experimental design was used to compare age-adjusted adult hospitalization rates before UBP initiation (2007-2010) against rates after UBP initiation (2011-2014) in San Diego County versus the rest of California.

 Difference-in-differences (DID) logistic regression models were estimated using hospitalization data from the California Office of Statewide Health Planning and Development for 2007 to 2014, including 372,205 and 642,455 hospitalizations for heart attacks and strokes, respectively.

 In the UBP versus pre-UBP period, the odds of adults being hospitalized for a heart attack in San Diego County decreased (odds ratio [OR], 0.84), whereas the odds stayed the same for adults in the rest of California (OR, 1.00): DID ratio of OR, 0.84 (

 <.001). This relative decrease was equivalent to 2735 (or 16.5%) fewer hospitalizations, totaling $61 million (2014 dollars). No robust association was found between UBP implementation and hospitalizations for strokes.

 A countywide physician organization learning collaborative was associated with fewer hospitalizations for heart attacks, but not for strokes. Healthcare systems and physician organizations should consider forming collaboratives to share best practices to manage patients’ cardiovascular and cerebrovascular risk factors, which may lead to fewer hospitalizations and reduced healthcare costs.

This study evaluated the California Right Care Initiative’s University of Best Practices (UBP) in

UBP is a countywide physician organization learning collaborative that shares best practices in managing cardiovascular and cerebrovascular risk factors. 

UBP was associated with lower age-adjusted hospitalization rates for heart attacks in San Diego County compared with the rest of California. 

Healthcare systems and physician organizations should consider forming collaboratives to share best practices to manage patients’ cardiovascular and cerebrovascular risk factors, which may lead to fewer hospitalizations and reduced healthcare costs.

Cardiovascular disease is the leading cause of death in the United States, and its annual costs, combined with those of stroke, averaged $316.6 billion per year in 2011 and 2012, including $193.1 billion in direct healthcare costs and $123.5 billion in indirect costs.

 Achieving lower heart attack and stroke rates will not only save lives, but also generate substantial healthcare cost savings.

Quality improvement collaboratives have become a well-established approach for improving the quality of care in the United States.

 Several national and regional models have emerged to reduce cardiovascular and cerebrovascular diseases and their risk factors. Million Hearts, for instance, an initiative launched by HHS in 2011, aims to prevent 1 million heart attacks and strokes over 5 years by convening and educating communities, physician groups, federal agencies, and other partners.

 Regional models that aim to reduce cardiovascular and cerebrovascular risk factors include physician collaboratives focused on education, training, and data sharing. For example, South Carolina’s Hypertension Initiative, which spread to 4 states in the stroke belt, contributed to a 43% decline in coronary heart disease deaths and a 42% decline in stroke deaths between 1996 and 2006.

 Another collaborative in Ontario, Canada, promoted education in 11 primary care clinical practices, which resulted in reductions in patients’ systolic blood pressure and 10-year cardiovascular risk scores during the 3-year study period.

 Finally, a community-wide cardiovascular disease prevention collaboration in rural Franklin County, Maine, was associated with lower hospitalization and mortality rates over a 40-year period.

The California Right Care Initiative (RCI) began in 2007, focusing on evidence-based and outcomes-improvement strategies to reduce preventable morbidity and mortality among Californians, particularly those with cardiovascular and/or cerebrovascular disease.

 One of RCI’s main strategies is to stimulate the sharing of best practices for managing risk factors (including diabetes) for these diseases among physician organizations, integrated delivery systems, and other stakeholders through regional inter-organizational learning communities known as University of Best Practices (UBP).

The San Diego County UBP (now called “Be There San Diego”) is the longest running RCI learning collaborative. San Diego County’s efforts began with a National Institutes of Health Grand Opportunity grant in October 2009, in which the California Department of Managed Health Care and physician leaders began to plan for the learning collaborative. The UBP meetings began in February 2011, and the monthly learning collaborative is attended by medical, pharmacy, and quality improvement directors from all of the major physician organizations, integrated delivery systems, and community clinics in the county. The attendees represent organizations that serve approximately 80% of San Diego County patients, including those who are commercially insured, Medicare, Medi-Cal, safety net, US Navy, and US Department of Veterans Affairs. Each meeting is devoted to presentations and discussions of better ways to care for patients with cardiovascular and cerebrovascular disease risk factors (eg, high blood pressure, high lipid levels, high blood sugar levels, diabetes, obesity, and smoking) through patient activation methods, healthcare team coordination, and uptake of medication protocols. Each organization reports and shares its progress on these measures because controlling these intermediate outcomes prevents unnecessary heart attacks and strokes.

The objective of this study is to examine whether the California RCI’s UBP in San Diego County was associated with fewer hospitalizations for acute myocardial infarctions (hereafter “heart attacks”) and cerebrovascular events (hereafter “strokes”). Learning collaboratives to improve coordination and quality of care are becoming more prevalent, as evidenced by recent initiatives including the CMS State Innovation Model Initiative, CMS’s Transforming Clinical Practice Initiative, the Medicaid program, and the Children’s Health Insurance Program.

 In spite of large-scale initiatives using regional learning collaboratives, no study has examined the impact of a physician organization learning collaborative in the context of a highly competitive managed care market that is attempting to collaborate on shared community-wide goals to better manage cardiovascular and cerebrovascular risk factors.

Inpatient hospitalization data are from California Office of Statewide Health Planning and Development’s (OSHPD) nonpublic Patient Discharge Data for 2007 to 2014 for adults 18 years or older. OSHPD collects these data from all hospitals in California. To identify relevant hospitalizations, we examined the principal discharge diagnosis code, which was based on the 

International Classification of Diseases, Ninth Revision, Clinical Modification 

). We used code 410 to capture acute myocardial infarction (heart attack)

 and codes 430 to 438 to capture strokes.

 Hospitalizations were converted to age-adjusted rates, based on California’s 2014 population,

 using the direct standardization method for those aged 18 to 64 years and those 65 years or older.

During the 2007 to 2014 study period, there were 25,287,552 hospitalizations in California, including 437,774 for heart attacks and 667,776 for strokes in acute care hospitals. To avoid counting multiple hospitalizations for the same event, we excluded hospitalizations when a patient was discharged to the admitting or another hospital for acute inpatient care. These transfers likely occurred because the patient needed care at a higher-acuity hospital. This reduced the number of hospitalizations to 372,205 (—15.0%) for heart attacks and 642,455 (–3.8%) for strokes.

To estimate whether the California RCI’s UBP in San Diego County was associated with changes in hospitalization rates for heart attacks and strokes, we estimated difference-in-differences (DID) models, which are used in quasi-experimental research designs.

 A DID model reduces the potential for bias by controlling for baseline differences in hospitalization rates between San Diego County and the rest of California, and by controlling for hospitalization rate reductions that occurred in the whole state during this entire period. Equation 1: 

shows the logistic regression model used to estimate the DID models, where 

is 1 for a hospitalization for individual (

is 1 for a hospitalization in San Diego County and is 0 for a hospitalization in other counties; 

is 1 for years 2011 to 2014 and is 0 for years 2007 to 2010; 

are time-varying, county-level control variables from the California Health Interview Survey,

 including the percentage of the adult population that did not have health insurance, the percentage that did not have a usual place to go when sick or needing health advice, and the percentage that had an income from 0% to 199% of the federal poverty level because these factors are associated with the ability to pay for healthcare services and access to care. The DID interaction term compares how the odds of a hospitalization changed in the UBP period (2011-2014) versus the pre-UBP period (2007-2010) for San Diego County residents versus the residents in the rest of California. Separate models were estimated for heart attack and stroke hospitalizations.

In addition to the UBP in San Diego County, Sacramento and Los Angeles counties began UBPs in July 2012 and April 2013, respectively; however, they have far less physician organization participation and lower fidelity of implementation of learning collaborative models.

 As a sensitivity analysis, we re-estimated the DID models by excluding these counties.

We also analyzed pre-UBP hospitalization trends for San Diego County and the rest of California, because a DID model assumes parallel pretreatment trends, which means that absent the UBP intervention, the hospitalization rates in San Diego County and the rest of California are expected to change at the same rate.

 As a supplemental analysis, we extended the pre-UBP analysis by examining pre-UBP versus UBP period hospitalization trends in San Diego County and the rest of California using a piecewise linear regression difference-in-differences-in-differences (DIDID) model. The 

Our data only include individuals who were hospitalized. Therefore, we estimated the logistic regression models using Stata version 12 (StataCorp; College Station, Texas) 

command that implicitly includes individuals who were not hospitalized, because one variable within the 

command is the population, which we obtained from the California Department of Finance.

 The number of individuals who were not hospitalized equals the population minus the number of hospitalizations from individuals who were hospitalized, which allows for more than 1 hospitalization in a year for a separate medical event for an individual (eg, 

 code 412 is for previous heart attacks, which we did not include). The other key variable within the 

command is the age-adjusted number of hospitalizations, which we calculated by multiplying the aged-adjusted rate by the population for each county(s)-year. The 

command produces the same results as if we had estimated our models as logistic regressions with individual-level data that include both hospitalized and nonhospitalized individuals.

Our blogit model estimation approach is similar to 2 studies that analyzed emergency department (ED) visits, which explicitly included individuals who were not present in the ED visit data.

The University of Best Practices physician organization learning collaborative in San Diego County was associated with lower hospitalization rates for heart attacks.

Hector P. Rodriguez, PhD, MPH

Articles

Implementation Variation in Natural Experiments of State Health Policy Initiatives

Organizational Influences on Healthcare System Adoption and Use of Advanced Health Information Technology Capabilities

Countywide Physician Organization Learning Collaborative and Changes in Hospitalization Rates