Currently Viewing:
The American Journal of Accountable Care September 2019
Achieving a Culture of Health: Steps for Engaging State Government
Dennis P. Scanlon, PhD; Jocelyn M. Vanderbrink, MHA; Mark Sciegaj, PhD; and Brigitt Leitzell, MS
Currently Reading
Implementation Variation in Natural Experiments of State Health Policy Initiatives
Diane R. Rittenhouse, MD, MPH; Aryn Z. Phillips, MS; Salma Bibi, MPH; and Hector P. Rodriguez, PhD, MPH
Building Trust Can Improve American Healthcare
Richard J. Baron, MD, President and Chief Executive Officer, American Board of Internal Medicine and the ABIM Foundation
The Case for Decoupling Chronic Disease Care From Insurance and Restructuring How We Pay for It
Jackson Williams, JD
The Cost of Learning: Participating in APMs Despite Projected Financial Losses
Jessica Walradt, MS; Hannah Alphs Jackson, MD, MHSA; Brian Walsh, BA; and David Manning, MD

Implementation Variation in Natural Experiments of State Health Policy Initiatives

Diane R. Rittenhouse, MD, MPH; Aryn Z. Phillips, MS; Salma Bibi, MPH; and Hector P. Rodriguez, PhD, MPH
This paper presents a method to characterize policy implementation across states to enable more nuanced impact assessments of federal healthcare delivery system and payment reforms.

Objectives: An increasing number of federal initiatives allow states flexibility in selecting the strategies used to achieve initiative-specific goals. Variation in the foci and intensity of implementation may explain why federal policy initiatives succeed in some states and fail in others.  The CMS State Innovation Models (SIM) initiative is a complex policy intervention implemented with substantial variation across states and may have variable impacts. This paper presents a method to characterize and account for that variation in states’ implementation foci and intensity in natural policy experiments.

Study Design: A combination of quantitative and qualitative measures of SIM implementation was used to characterize the foci of payment and delivery system reforms across states.

Methods: A modified Delphi expert panel process was used to prioritize the features of SIM implementation that would differentiate grantee states with respect to improved health outcomes. Three researchers then reviewed summaries of published evaluations and reports to characterize and score states on each implementation feature. Expert panelists guided the researchers on developing the criteria and weights applied to the focus areas when calculating SIM implementation intensity scores for states.

Results: Over 3 years of an expert panel process, 4 dimensions of SIM implementation that would most affect health outcomes were prioritized: (1) extent and breadth of stakeholder engagement, (2) extent that SIM implementation was focused on improving behavioral health, (3) amount of SIM funding per capita, and (4) breadth and depth of value-based payment reforms. Scoring states based on the prioritized factors resulted in composite scores that differentiated states into 3 categories: high, moderate, and low implementation intensity.

Conclusions: We developed a stakeholder-driven method to measure and account for variation in implementation foci and intensity in a federal policy initiative that was implemented heterogeneously across grantee states. Our method for characterizing state implementation variation may be useful for natural policy experiments examining the variable impact of policy initiatives.

The American Journal of Accountable Care. 2019;7(3):12-17
Over the past decade, health policies and programs intended to spur innovation in delivery system design and payment reform have become commonplace across the United States.1 Studies examining the effect of state health policies rely on natural experiment study designs, but they do not account for differences in states’ foci and experiences of policy implementation. Characterizing states as exposed or not exposed (1 or 0), as is traditionally done in natural experiments of state health policy initiatives, is overly simplistic and does not consider the specific strategies used by states. Ideally, features of each state’s rollout, including reform foci and intensity of activities, could be modeled quantitatively. The small number of states involved in any given reform, however, precludes the use of quantitative methods to produce a taxonomy to characterize “types” of policy implementation using k-means cluster analysis or another data reduction method.2 As part of a natural experiment of the federal–state program—the CMS State Innovation Models (SIM) initiative—we describe a stakeholder-driven method to prioritize, assess, and account for state-level variation in natural policy experiments.

The SIM initiative awarded funding and technical assistance to states through a competitive process. State health departments proposed plans to implement innovative delivery and payment models to improve health system performance, improve the quality of patient care, and decrease healthcare costs for all residents of the state. Through SIM, the federal government provided states with more than $1 billion in funding and substantial technical assistance to plan, pilot test, and implement payment and delivery system reforms.3 Round 1 of SIM funding was awarded in April 2013 to 6 states (Arkansas, Maine, Massachusetts, Minnesota, Oregon, and Vermont). Round 2 was awarded in December 2014 to 11 additional states (Colorado, Connecticut, Delaware, Idaho, Iowa, Michigan, New York, Ohio, Rhode Island, Tennessee, and Washington). Some states that applied for, but did not receive, SIM funding were awarded modest planning grants ($3 million or less) to aid in advancing their innovations to the potential testing phase in the future. This staged roll-out of SIM allows for a natural experiment study design to evaluate the impact of this policy on population health outcomes.

Previous reviews have conceptualized the critical role of variation in implementation processes to understand differential impacts of policy change.4,5 Implementation science considers intensity and other aspects of the implementation process, including the adoption, reach, and fidelity of implementation to intended policy features.6 The application of implementation science in health services and policy research is growing, but it primarily focuses on the ways in which practitioners successfully incorporate new policies into routine practice as study outcomes.7 Studies have rarely examined how federal policies are differentially implemented at the state level and how these variations affect healthcare utilization and health outcomes.8

The political science subfield of policy implementation research analyzes sources of variation in the implementation of large-scale policies (ie, laws and regulations) and does consider policy goals such as health outcomes as dependent variables, but as with the other perspectives, it does not study how the variation itself influences these outcomes. A handful of policy implementation research studies have described variation in the focus of state-level policy implementation, including applications to welfare policies, medical marijuana policies, and youth sports traumatic brain injury policies.9-12 However, we could find no empirical studies that simultaneously characterized the foci and intensity of state-level policy implementation—considerations that are critically important for understanding the impacts of a complex, multifaceted policy intervention like SIM.

Our conceptualization of the connection between policy implementation and outcomes is most similar to that of Strehlenert and colleagues’ Conceptual Model for Evidence-Informed Policy Formulation and Implementation,5 which covers the entire policy process from agenda setting and policy formulation to implementation and outcomes evaluation; however, this framework was used only descriptively with case studies and not to make comparisons across multiple implementers. CMS allowed states considerable latitude in SIM plan foci and implementation strategies,13 and this variation in policy implementation could result in differential impacts of SIM on utilization and health outcomes across the grantee states. To advance the examination of heterogeneous effects in natural policy experiments, we developed a stakeholder-driven method to measure and account for variation in implementation foci and intensity in a federal policy initiative that was implemented heterogeneously across states.


We used a combination of quantitative and qualitative measures to prioritize, classify, and analyze SIM implementation variation for each of the 17 grantee states. To do this, we convened an expert advisory panel composed of 8 SIM leaders from different states to provide us with qualitative and quantitative input about core SIM activities and, ultimately, to participate in a modified Delphi expert panel process to prioritize key differences in implementation foci and strategies across the SIM states. The panel members were recruited from 8 SIM grantee states: Arkansas, Colorado, Iowa, Oregon, Maine, Michigan, Minnesota, and Washington.

Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up