 In 2009 and 2010, 17 primary care sites within 1 healthcare system became patient-centered medical homes (PCMHs), but the sites trained different personnel (pharmacists vs nurses) to improve diabetes care using self-management support (SMS). We report the challenges and successes of our efforts to: 1) assemble a new multipayer (Medicare, Medicaid, commercial) claims dataset linked to a clinical registry and 2) use the new dataset to perform comparative effectiveness research on implementation of the 2 SMS models. 

 We lost permission to use private-payer data. Therefore, we used claims from Medicare fee-for-service and Medicare/Medicaid dual-eligible patients merged with chronic disease registry data. We studied 2008 to 2010, which included 1 year pre- and 1 year post the 2009 implementation time period. Outcomes were outpatient and emergency department visits, hospitalizations, care process (use of statin), and 3 intermediate outcomes (glycemic control, blood pressure [BP], and low-density lipoprotein cholesterol [LDL-C]). 

In our sample of 2826 patients, quality of care improved and utilization decreased over the 2.5 years. Both approaches improved lipid control (LDL-C decreased by an average of 4 mg/dL for pharmacy-SMS and 5.6 mg/dL for nurse-SMS) and diastolic BP (—1.5 mm Hg for pharmacy-SMS and –1.3 mm Hg for nurse-SMS), whereas only the pharmacy-led approach decreased primary care visits (by 0.8 visits). The groups differed slightly on 2 measures (glycated hemoglobin, systolic BP) with respect to the trajectory of improvement over time, but performance was similar by 2.5 years. 

 Diabetes care improved during PCMH implementation systemwide, supporting both nurse-led and pharmacist-led SMS models. 

We compared 2 chronic disease self-management support approaches (pharmacist- vs nurse-led) within 1 healthcare system over 2.5 years. 

Both approaches improved lipid control and diastolic blood pressure.

Only the pharmacy-led approach decreased primary care visits. 

When examining the trajectory of change over 2.5 years, there were small differences in the trajectory of glycated hemoglobin and systolic blood pressure change favoring the nurse-led approach. However, both differences could be explained by preexisting differences in patient populations prior to patient-centered medical home implementation. By study end, there was no difference in either measure. 

Therefore, we conclude that both approaches can improve the quality of diabetes self-management support.

The American Recovery and Reinvestment Act (ARRA) invested more than $1 billion in redesigning healthcare delivery systems.

 In response to interest in using electronic health records (EHRs) to perform comparative effectiveness research (CER) on chronic disease management,

 we were funded by ARRA to develop a unique database that linked a longitudinal chronic disease registry database to multipayer claims from Medicare, Medicaid, commercial, and county insurance plans. The chronic disease registry, originally designed to support the quality metrics required by the various payers, contained care process measures and some intermediate outcomes. It also had the advantage of using a physician-adjudicated process to confirm patient inclusion. Our goal was to create a large relational database with the flexibility to allow construction of longitudinal datasets to answer specific questions about quality and utilization. The power of such a database would facilitate research that compares care among clinical sites and payer types and examines differing approaches to care.

As a proof of concept, we aimed to test for measurable improvement over time in diabetes care quality and utilization during the implementation of patient-centered medical homes (PCMHs) in the University of Michigan Health System (UMHS) in 2009, including 1 year pre- and 1 year post implementation (resulting in the study window 2008-2010). Of the conditions captured by our registry (including diabetes, heart failure, coronary artery disease [CAD], and asthma), we chose to study diabetes, the largest and most well-established disease registry. Prior research on PCMHs has shown inconsistent benefits from PCMH efforts, such as improved quality or utilization of healthcare,

 so we used this unique opportunity to measure the longitudinal improvement in individual patients’ quality of diabetes care before, during, and after PCMH implementation.

 UMHS recognized the importance of self-management support (SMS) in accomplishing 2 principles, whole-patient and multidisciplinary care. UMHS devoted the most resources to developing SMS for diabetes, a high-priority condition. 

All sites were operating under a common administration and therefore were unified under the same EHR, staffing formularies, quality metrics, chronic disease registries, and efforts to add extended hours and improve communication with patients. Personnel from all participating sites underwent the same quality improvement to improve diabetes management SMS care, which included education sessions, training in the use of laminated cards, standing lab-order sets, note templates, patient handouts, a database, and flow sheets to track improvement. However, there was lack of consensus among the sites regarding which type of staff to train: some sites chose to train pharmacists, while others felt that a clinical nurse coordinator model would be more effective. Both SMS models have been shown, separately, to improve diabetes care quality,

 making SMS a reasonable question for CER. 

In this article, we report the challenges and successes of our experience with assembling and using the new multipayer diabetes dataset, and we perform CER on PCMH and 2 SMS models implemented within a single health system. We hypothesized that there would be an overall improvement over time in the diabetes care process and intermediate outcomes (eg, glycemic control) and declining utilization of outpatient and inpatient visits, but that there would be no difference on any of the measures between the 2 SMS models. 

UMHS is a large multisite health system consisting of a main university hospital and 4 specialty hospitals. UMHS provides ambulatory care to more than 220,000 established patients. At the time of the PCMH implementation in 2009, there were 17 primary care sites; as of 2017, there are 27. 

In 2005, the UMHS faculty group practice developed chronic disease registries for several diseases: diabetes, heart failure, asthma, and CAD. These registries were maintained by the UMHS Quality Management Program (QMP), which was responsible for determining patient eligibility for the chronic disease registries by triangulation of patients’ problem lists, laboratory test results, and medications relevant to that particular condition. 

For entry into the diabetes registry, a patient needed to have 2 outpatient visits (with primary care or endocrinology) or 1 hospitalization or emergency department (ED) visit in the past 3 years with a billing diagnosis of diabetes, which was validated by evidence of a diabetes medication (eg, insulin), diabetic supply (eg, glucometer), or glycated hemoglobin (A1C) greater than 6.4%; excluded were gestational diabetes and steroid-induced diabetes. Quality reporting was limited to those patients considered active, defined as 2 ambulatory visits within the past 2 years and 1 visit within the past 13 months.

 This method allowed an unbiased assessment of care in which providers cannot selectively enter patients into the registry. 

All registries were composed of longitudinal semi-annual data calculated on June 30 and December 31 reflecting whether recommended healthcare was provided in the prior 12 months. This enabled the QMP to provide health plans with longitudinal reports regarding the quality of care provided to their patients. 

In addition to maintaining the chronic disease registries, the QMP also submitted supplemental clinical data to health insurers and received multipayer claims to facilitate quality measurement and improvement. In 2010, the QMP was actively collecting ambulatory and hospital claims data for patients who selected a UMHS physician (eg, a managed care plan) or were attributed to a UMHS physician (eg, fee-for-service [FFS] plans) from 4 payers: 1) Medicare FFS, 2) dual-eligible Medicare/Medicaid, 3) a commercial plan providing Medicaid managed care, and 4) a large commercial insurer in Michigan offering a variety of levels of coverage. 

By the start of this project, we had permission from these 4 payers to merge claims data with care process measures into 1 relational dataset for future health services research. The Medicare FFS data and agreement to use the data for research were part of a Data Use Agreement (#23675) with CMS for the Physician Group Practice Medicare Demonstration Project

 and approved for human subjects research at the University of Michigan (#HUM00041118).

From the QMP diabetes registry, we selected only patients 51 years or older. Although quality measures are applied to patients aged 18 to 75 years,

 we limited the data to patients 51 years or older due to sparse data for younger patients. In addition, prior study results have shown that despite age limits on quality measures, provision of the care does not stop at age 75.

 Because there was no upper age limit for inclusion in the QMP registry, we included patients older than 75 years for whom no quality measures for diabetes are currently defined. 

In 2008, in response to nationwide efforts to improve health system performance, UMHS began developing PCMHs at each of its 17 primary care sites. By 2009, all sites were certified as PCMHs by Blue Cross Blue Shield of Michigan criteria.

Given the option of selecting either pharmacists or nurses to be trained in improved SMS for the care of diabetes, the sites were divided. Twelve sites chose the pharmacist model (pharmacy-SMS) and were, in general, the internal medicine practices, whereas the 5 sites that chose the nurse-coordinator model (nurse-SMS) were, in general, the family medicine practices. Pediatrics-only (another 9 sites) and geriatrics-only clinics (1 site) did not participate. Patients could have primary care at only 1 site. 

We considered 3 utilization outcomes (ED visits, outpatient visits, and inpatient visits) and all quality measures collected by the QMP. Utilization of each type was expressed as number of visits over the past 12 months tallied repeatedly at the end of each 6-month period. For the care of diabetes, we decided to use 4 of the 7 collected quality measures: 1 care process variable (whether or not a statin was ordered, as a dichotomous outcome) and 3 intermediate clinical outcomes (glycemic control [A1C], in % points; blood pressure [BP], systolic and diastolic, in mm Hg; and low-density lipoprotein cholesterol [LDL-C], in mg/dL, all as continuous variables). For each of five 6-month reporting periods during the 2.5 years, the quality measures were calculated using the most recent data up to 12 months, resulting in 5 time points. 

 tests to compare patient characteristics between the 2 SMS types, using the baseline sample of patients at time point 1. We used 

 tests to test for adjusted simple temporal trends in the outcomes over the 2.5-year study. 

Implementation of the PCMH at each of the 17 sites took place during time points 2, 3, and 4, over varying months that could span more than one 6-month period. Because we were unable to designate a single point in time when we could directly assess a change in slope, we assessed the mean slope in outcomes across the 5 time points, where time point 1 assuredly occurred before implementation of the PCMH at all sites and time point 5 assuredly occurred at least 6 months after the implementation of the PCMH at all the clinical sites. To analyze each outcome measure, we used multilevel regression (logistic for statin use; linear for the continuous utilization and intermediate outcomes), including a random intercept for each patient. Our predictors were SMS type (pharmacy-SMS vs nurse-SMS), time (in 5 half-year increments), quadratic transformation of time (time-squared, which tests for curvilinear trajectories), and time interactions with SMS type (SMS type multiplied by both the linear and quadratic time term). Multilevel regression allows use of data from all patients, even those who entered late, exited the dataset early, or had missing time points. Because patients who utilize less ambulatory care generally are delivered less recommended care,

 dropping patients with incomplete data would be expected to result in biased estimates for the QMP measures. Using this method, the result of the time variable indicates mean change in each outcome per time period (ie, the slope over time) for the entire sample, the SMS-type variable indicates the overall mean difference in the outcome between the 2 groups, and the time-SMS interaction term indicates the overall difference in slope over time between the 2 groups. 

Patients seeking care at the pharmacist-SMS sites were more complex (had a higher comorbid condition count) than those at the nurse-SMS sites. Because a more complex patient population

 might facilitate providing better diabetes care,

 we performed a propensity score model. We calculated inverse probability weights

 for each patient based on age, gender, and comorbid condition count to adjust for each patient’s likelihood of seeking care at either of the 2 types of SMS sites, then applied the weights as controls to the final multivariable models.

During 2010, UMHS cared for more than 12,000 established patients with diabetes (

) from any source of all-payer claims data. We constructed all utilization and comorbidity variables across five 6-month time periods from July 2008 to December 2010. We then merged claims data from 4 payers with which we had agreements (Medicare FFS, Medicare/Medicaid dual-eligible, commercial-Medicaid managed care, and a large commercial plan) with chronic disease registry data by patient and time period. This resulted in a merged dataset with 9637 unique patients with diabetes (Figure 1). 

However, in 2012, the commercial insurers decided to limit any reporting of results of our analyses to quality improvement reporting. We had an executed Data Use Agreement (DUA) with Medicare, as well as University of Michigan Institutional Review Board approval, so we were able to report analyses using Medicare FFS and dual-eligible patients. When we restricted the study group to Medicare, we included 2826 unique patients in Medicare FFS and dual Medicare/Medicaid (Figure 1). After restricting the data to those 51 years and older, the number of patients per time period ranged from 2221 in the first time point to 1701 in the fifth time point. 

There were a few notable differences in baseline demographic and clinical characteristics of patients in the 2 SMS models (

). Patients in the pharmacy-SMS were sicker (2.6 vs 2.3 chronic conditions) than those in the nurse-SMS. The nurse-SMS sites started with higher mean BP of patients at baseline (Table 2) (133/70 vs 129/68 mm Hg; 

<.001) and the patients used more primary care (5.7 vs 5.3 annual visits; 

<.03), whereas the pharmacy-SMS clinics’ patients used more specialty care (4.9 vs 3.2; 

<.001) and had more hospitalizations (0.55 vs 0.39; 

Over the duration of the study (2010 vs 2008, which includes implementation of the PCMHs), there were significant (

<.05) downward trends in unadjusted LDL-C (Table 1) (—4 mg/dL for pharmacy-SMS vs –5.6 mg/dL for nurse-SMS), diastolic BP (–1.5 mm Hg for pharmacy-SMS, –1.3 mm Hg for nurse-SMS), and utilization of primary care (0.8 office visits for pharmacy-SMS only). 

When we measured nonlinear trajectories across 2008 to 2010 (including PCMH implementation during 2009) between the 2 models, however, we found small, but statistically significant, differences in quality of care trajectories between the groups. The nurse-SMS patients had an initial increase (ie, positive slope from prior to during PCMH implementation) followed by a decrease in A1C after PCMH implementation, resulting in a net 0.1% improvement. For pharmacy-SMS, the patients’ A1C remained flat at 7.1% (

<.001 for the difference in time trajectories between SMS types) (

). Second, systolic BP started higher in the nurse-SMS patients (133 vs 129 mm Hg) (Table 2) and had a steeper decline in the early period (

When we applied propensity scores (inverse probability weights based on comorbid condition count, age, and gender) to the final multivariable models, there were no changes in any of the results concerning differences in change over time by SMS type. 

The use of EHR and registry data merged with administrative claims across healthcare sites and payers has been championed as a hallmark of modern patient-oriented CER.

 We were ideally poised to assemble a longitudinal dataset for such use during a 2.5-year period that spanned time pre- and post implementation of a PCMH model across multiple clinical sites in a large healthcare system, with sites differing only by type of SMS: pharmacy-based versus nurse-based SMS. However, our efforts were challenged by changes in the private sector with regard to using claims data for research. Despite the challenges, we were able to use Medicare and dual-eligible data (CMS DUA#23675). We found modest longitudinal improvements in unadjusted LDL-C (4 and 5.6 mg/dL for pharmacy- and nurse-based SMS, respectively) and diastolic BP (1.5 and 1.3 mm Hg for pharmacy- and nurse-based SMS, respectively), thus supporting SMS in general. When comparing the trajectories of the 2 SMS models across the 2.5-year study period, the clinical outcomes were similar. We found small differences in the rate of improvement of glycemic control and systolic BP that modestly favored the nurse-based SMS model, possibly due to a poorer baseline performance (therefore they had more room to improve more quickly). Both groups decreased annual utilization of primary care, but the pharmacy-based SMS group significantly decreased utilization by 0.83 visits compared with their pre-implementation rates. 

Our findings are consistent with prior study results on PCMHs overall that found little effect on clinical outcomes,

 including a study that also used longitudinal methods.

 Specifically regarding diabetes care, our results are consistent with a recent meta-analysis of 48 PCMHs, which found average decreases in LDL-C of 3.9 mg/dL and of 1.5 mm Hg in diastolic BP.

To our knowledge, ours is the first study to conduct a concurrent comparison of 2 different SMS models within otherwise similar PCMH models within 1 health system.

 Prior studies have grouped the SMS models, whether nurse- or pharmacist-led, into promotion of self-management or team changes.

 Overall, we found statistically significant, but not substantial, differences between SMS models. Slightly poorer baseline values among the nurse-SMS patients likely contributed to greater room for improvement, a finding also observed in meta-analysis.

 It is important to note that improved intermediate measures (LDL-C, A1C, BP) suggest improved cardiovascular risk profiles, but finding cardiovascular benefits would require longer follow-up to realize, a larger sample size of sites beyond our system, and more aggressive interventions.

Because the patients attributed to the 2 SMS models also differed by medical complexity, the small differences in quality we observed could also have been due, in part, to patient-level differences. Patients with more complex comorbid condition burden are more likely to receive recommended healthcare.

 We lacked the sample size (number of sites) and variation to be able to formally test for confounding by comorbidity, however, so we applied a propensity score by patient as an alternative approach. 

There are several limitations to this study. First, we lacked a concurrent control group (ie, non-PCMH patients), which decreased our ability to detect differences. Second, because we only had 2 sources of payer data, we cannot generalize our findings across multiple payer sources. Third, clinical outcomes (glycemic control, BP, and lipid control) were calculated only among those monitored for the outcomes (ie, a patient without a lipid level could not be included in the lipid control outcome). When 1 measure depends on the proper performance of an upstream variable, biases in performance can be introduced. One approach is to give a failing score for the downstream performance measure if the earlier process is not performed,

 which would provide additional incentive to perform the upstream care. 

Despite the limitations, our study had certain strengths. We used multiple snapshots across time, which can capture differences in trajectories in addition to overall differences over time. We harnessed 17 sites, garnering a larger sample than would be possible within 1 clinical quality improvement site. 

We faced a key barrier to performing multipayer health services research: loss of permission to use private-payer claims data for research. Until merged datasets can be constructed with equivalent health services variables and durable continuing permission to perform research, it will be difficult to efficiently perform CER across or between payers. Future research in comparing chronic care delivery approaches may be best served in large single-payer system with a diversity of patient complexity, such as the Veterans Affairs Healthcare System.

Articles

WEB EXCLUSIVE

In this longitudinal comparative effectiveness study of different chronic disease self-management support approaches within 1 system, both pharmacist- and nurse-led patient-centered medical home approaches improved diabetes care.

Diabetes Care Improvement in Pharmacist- Versus Nurse-Supported Patient-Centered Medical Homes

Although hospitals face increasing pressure from payers to improve the efficiency of healthcare delivery beyond the index hospitalization, they often lack information on postdischarge events. The Michigan Value Collaborative (MVC) developed a claims-based algorithm to provide hospitals with data on events that occur to patients beyond the hospitalization. Herein, we discuss the validation of MVC’s claims-based algorithm. 

Retrospective analysis of a claims-based algorithm’s ability to identify specific medical events, such as index hospitalizations, 30-day readmissions, emergency department visits, skilled nursing facility admissions, home health visits, and rehabilitation services. The claims-based events were validated using a primary review at 63 hospitals. 

 We selected 1830 Blue Cross Blue Shield of Michigan episodes from MVC data and asked 63 Michigan hospitals to query their medical records for the presence or absence of specific events. We then calculated agreement statistics and improved our algorithm using feedback from hospitals. 

 All 63 hospitals participated in the validation process and successfully identified 99% of episodes in their medical records. The initial agreement between our algorithm and medical records was moderate for 4 postdischarge events (kappa ranging from 0.62-0.78) and poor for rehabilitation services (0.16). Much of the disagreements occurred because hospitals could not identify postdischarge events occurring outside of their hospital systems. Other disagreements occurred because of hospital coding practices. Through this analysis, the claims-based algorithm was improved to better reflect real-world coding practice.

 Our findings suggest that the MVC claims-based algorithm identifies and classifies claims with high fidelity and outperforms medical records in the identification of postdischarge events. These findings provide important insight to policy makers, payers, and hospital administrators about the value of claims-based data for the implementation of episode-based programs.

The Michigan Value Collaborative (MVC)’s claims-based algorithm identifies postdischarge events with high fidelity, often outperforming medical records.

The creation of this algorithm and its subsequent validation demonstrates that deriving episode-level utilization from administrative claims is achievable. 

With an increasing focus on improving efficiency of care after the index hospitalization, hospitals need accurate information. To this end, MVC’s claims-based algorithm provides precise episode-level data on postdischarge care. 

Given that administrative claims data outperformed medical records, hospitals should seek to collect such data, possibly by participating in a statewide, regional, or health-system value improvement collaborative or asking payers for claims-based analytic data.

Hospitals are increasingly being held accountable for services and expenditures that occur beyond the hospitalization through episode-based performance measures.

 Postdischarge expenditures, such as postacute care and readmissions, have been cited as the fastest growing spending categories over the last 2 decades and have been the target of many national programs focused on reducing healthcare costs.

 For example, CMS recently implemented the Comprehensive Care for Joint Replacement bundled payment program, which will hold hospitals financially accountable for expenditures occurring from admission through 90 days post discharge.

 In addition, accountable care organizations were developed to reduce costs that occur both outside of and during hospitalizations.

 Based on the prevalence and growth of episode-based payment programs, it is evident that many payers believe the key to reducing healthcare expenditures is to hold hospitals responsible for efficiency along the entire patient care episode. 

Despite enthusiasm for increased episode efficiency, identifying specific high-cost events, such as readmissions, can be challenging for hospitals for several reasons.

First, it is difficult for hospitals to track events outside of the initial hospitalization. In fact, hospitals are often not even aware of postdischarge events that occur at outside facilities. Secondly, unless directly affiliated with the hospital, postdischarge providers are not incentivized to report utilization patterns to hospitals. Furthermore, many small hospitals may not have internal resources to monitor and track postdischarge events and spending. 

In Michigan, one response to these challenges was the development of the Michigan Value Collaborative (MVC).

 Established in 2012 and funded by Blue Cross Blue Shield of Michigan (BCBSM), MVC’s mission is to provide hospitals with episode-level data and promote high-quality care at the lowest reasonable costs.

 One particular area of interest has been postacute care, specifically rehabilitation. MVC hospitals have begun to use episode-level data to monitor rehabilitation expenditures, especially for conditions such as acute myocardial infarction (AMI) and hip replacement, where more than one-third of all patients are discharged to rehabilitation facilities (

However, a necessary step in this process is to ensure that MVC’s claims-based identification of postdischarge services is accurate. 

In this context, we describe a large-scale medical records—based validation of the algorithm used by MVC to define clinical episodes of care in commercial claims. We believe that the MVC validation experience will provide useful insight to hospitals and payers about the advantages and limitations of using claims to track events that occur after hospitalization.

The MVC collects claims data for BCBSM beneficiaries admitted to 1 of 63 Michigan hospitals for 21 medical and surgical conditions. We consulted clinical experts who used 

International Classification of Diseases, Ninth Revision

) diagnosis/procedure and Current Procedural Terminology (CPT) codes to define the conditions. Then, we utilized variables such as revenue codes, diagnosis-related groups (DRGs), facility identification, and CPT codes to classify individual claims into: inpatient, skilled nursing facility (SNF), home health, emergency department (ED), inpatient rehabilitation, outpatient rehabilitation, and general outpatient claims. Cases with readmissions to hospitals other than the index facility were excluded from the readmissions validation step. 

) further details the entire attribution process. Table 1 presents the characteristics of episodes identified from claims data by condition type, including use of postdischarge services. This study was deemed exempt from review by the Michigan Institutional Review Board. 

Our validation process occurred in 2 phases (pilot and full validation). During the pilot, 6 hospitals were asked to review 10 to 20 BCBSM preferred provider organization (PPO) cases from between January 1, 2013, and October 31, 2014. For matching purposes, the clinical condition, national provider identifier, date of birth, gender, admission date, and discharge date for each patient was provided. Specifically, participants were instructed to indicate if the patient listed had records demonstrating that a specified event occurred within 90 days of their discharge date.

These events included 30-day readmissions, ED visits, SNF admissions, home health visits, and rehabilitation services (inpatient and outpatient).

The lessons learned from the pilot were used to inform the full validation. Here, we distributed the same key variables to all 63 MVC participants. Each hospital was provided 30 cases to review, with the exception of hospitals that participated in the pilot (which were asked to review 20 cases). We selected conditions based on volume, prevalence of associated postdischarge services, and suggestions from clinical experts. The conditions included colectomy, coronary artery bypass graft, AMI, pneumonia, congestive heart failure, hip replacement, knee replacement, cesarean delivery, vaginal delivery, trauma, and spine surgery. In total, 1830 BCBSM PPO cases were selected for data validation from these 11 conditions. 

We identified areas of agreement and disagreement between MVC’s claims-based algorithm and medical records. First, we looked for agreement that these episodes occurred and were attributed to the correct hospital and to the correct condition. Next, we used a kappa statistic to assess agreement for postdischarge services (

). For each disagreement in a case, 2 members of the MVC team, a clinician and an analyst, reviewed the specific claims. After determining the cause of the discordance, we adjusted our algorithm to better capture hospital events and re-evaluated the level of agreement. 

In the cases where MVC reported a postdischarge event that did not match a hospital’s medical records, we used confirmatory evidence from the claims to improve our confidence that the event occurred. Specifically, we examined the following items: 1) Did the postdischarge event have more than 2 claim line items with dates following the index admission?; 2) Did the place of service designation on the claims support the assignment of the claim to the postdischarge service?; and 3) Did the revenue center code on the claims support the assignment of the claim to the postdischarge service? 

One hundred percent of hospitals (n = 63) participated in the validation process. Hospitals matched 1812 of the 1830 (99%) MVC episodes to records in the hospital’s medical charts. 

The agreement for the occurrence of postdischarge services ranged from 0.16 to 0.78, with rehabilitation services having the poorest agreement. eAppendix B contains details of the validation process. 

Using administrative claims, we were able to identify 183 readmissions for 1812 episodes (10% readmission rate). Of these, there were 15 cases in which MVC observed a 30-day readmission that was not evident in the hospital data (

These were due to readmissions to a hospital different than the index facility and, per the methods section, were excluded. There were 24 discordant readmissions which MVC did not identify and a hospital did; these cases were classified as observation unit stays in MVC data (eAppendix B). 

We identified 452 ED visits (25% of episodes). Of these visits, 292 (65%) were identified by hospitals. Importantly, there were 160 out of 452 cases (35%) in which MVC observed an ED visit that was not reported in the hospital clinical data (Table 2). Of the 160 discordant cases, 122 (76%) of these ED visits occurred at a different facility than the index admission. All 160 cases had confirmatory evidence of a valid ED visit. 

There were 85 episodes (5%) in which a hospital reported an ED visit that was not evident in the MVC data. Upon review, many of these were classified elsewhere. For example, 38 preceded an index admission or a readmission and the ED services are grouped with this hospitalization in the MVC episode (eAppendix B). 

We identified 64 SNF admissions (4% of episodes). Of these, 43 (67%) admissions were identified by hospitals. There were 21 cases (33%) in which MVC identified a SNF admission but the admissions were not reported by hospitals (Table 2). We found evidence in the claims data to confirm that all 21 visits occurred. There were only 9 cases in which hospitals reported care in a SNF, and we found no evidence of a SNF admission in the MVC claims data (eAppendix B). 

We identified 949 home health visits (52% of episodes). Of these visits, 781 (82%) were identified by hospitals (Table 2). Of those reported by hospitals, there were 168 cases (21%) in which MVC observed a home health visit that was not evident in the hospital clinical data. There were 99 cases (10% of episodes) in which a hospital observed a home health visit that was not evident in the MVC data (eAppendix B). 

The algorithm identified 1223 rehabilitation visits (67% of episodes). Of these visits, 350  (29%) were reported by hospitals. There were 873 cases (71%) in which MVC observed a rehab visit that was not evident in the hospital clinical data (Table 2). Of the 873 discordant cases, 851 (97%) had confirmatory evidence in the MVC claims for utilization of rehabilitation services. There were only 42 cases (3% of episodes) in which a hospital observed a rehab visit that was not evident in the MVC data (eAppendix B). 

Improvements to the Claims-Based Algorithm 

After reviewing all cases of discordance between the claims-based algorithm and the medical records, we identified several areas for improvement, the most important being to update the 

 codes used to identify postdischarge claims to better reflect the coding practices used by hospitals. For example, 

 code V57.8 (care involving other specified rehabilitation procedure) was initially not considered by MVC as a related diagnosis code for patients after joint replacement. However, this code was used for a large number of rehabilitation services after joint replacement. After making this and other improvements, we re-evaluated the level of agreement between our algorithm and the medical records and found improvement in agreement for all services (eAppendix B). 

In this study, we validated the MVC’s claims-based algorithm for the identification and classification of postdischarge events. During the process, we found a high episode match rate between MVC data and medical records. Much of the disagreement was due to the inability of hospitals to identify readmissions, rehabilitation services, and other postdischarge events. Collectively, these findings suggest that this claims-based algorithm outperforms medical records in identifying postdischarge events. 

Previous investigators have convincingly demonstrated that variation in episode spending is largely due to postdischarge events.

 Preventable readmissions and variations in postacute care could indicate areas to improve hospital efficiency and outcomes. Others have also validated and demonstrated on a health-system level the success of claims-based algorithms in identifying hospital events.

 Existing literature has demonstrated the effectiveness of utilizing these tools to identify high-cost inpatient events and improve the value of care.

 It is reasonable to believe that by using this same approach to identify postdischarge events, institutions could potentially achieve similar results outside the hospital setting. The current study findings demonstrate, on a large statewide scale, that accurately identifying and measuring postdischarge utilization may be difficult for hospitals using medical records alone.

A claims-based algorithm could better identify postdischarge events, especially those that occur outside hospitals’ networks. With the current national focus on episode efficiency, identification of these events is imperative to driving high-value care. 

Our study has several limitations. First, we only included BCBSM patients in our validation process. Due to CMS privacy restrictions, we were unable to validate our algorithm with Medicare beneficiaries. Second, the MVC algorithm may not be generalizable for other commercial payers, although BCBSM is the largest commercial payer in Michigan. Third, we did not have hospitals look for readmissions that occurred outside of the hospital where the index event occurred. This decision was primarily made to reduce the chart review burden to hospitals; we received early feedback that hospitals could not identify these particular events. Finally, we did not validate our classification algorithm for other postdischarge events (eg, outpatient procedures) and intensity of services (eg, SNF length of stay). However, this study was focused on validating the occurrence of major postdischarge services. Although we did not end with perfect agreement between the MVC data and the medical records, there were only a few events identified by hospitals not seen in MVC claims (0.5%-4%). 

Our findings will help stakeholders understand the opportunities and challenges of using a claims-based algorithm to measure episode spending. Relevant to hospital administrators, the finding that the claims-based algorithm used in this study outperformed medical records suggests that such data provide more complete intelligence about the postdischarge period. This finding should encourage hospital administrators to obtain additional claims data by participating in a statewide, regional, or health-system collaboration and by asking payers to share these data. Without these claims data, hospitals will be limited in their ability to measure and optimize services provided outside of their facilities. This is particularly important as CMS and commercial payers are increasingly using episode-based performance measurement and payment bundling. 

Moving forward, research in this area should focus on how these data can be refined to provide more granular information to hospitals. For instance, providing hospitals with data on the average length of stay and intensity of services provided at SNFs may help providers understand the efficiency of facilities where patients are sent after discharge. Ultimately, the value of episode-based performance measurement and bundled payment programs as mechanisms to drive high-value care will strongly depend on the accurate measurement of episode-level payments and utilization.&ensp;

The authors thank Dr Vinita Bahl (University of Michigan), Kelly Rice (Beaumont Health System), Steve Lewis (St. Joseph Mercy Health System), Meghan Coughlin (MidMichigan Medical Center), and John Robertson (Hillsdale Community Health Center) for their comments and insight during the course of this research. They would also like to thank Dr David Share, Ellen Ward, Tom Leyden, and the Value Partnerships at BCBSM for their ongoing support of the Michigan Value Collaborative and this manuscript.

 Institute for Healthcare Policy and Innovation (CE, DCM, JMD), and Dow Division of Health Services Research, Department of Urology (CE, JDS, DCM, JMD), and Michigan Value Collaborative (CE, JDS, BV, VG, DCM, JMD), University of Michigan, Ann Arbor, MI.

 This research was supported by the Agency for Healthcare Research and Quality (1F32HS024193-01 to Dr Ellimoottil). Dr Miller receives salary support from Blue Cross Blue Shield of Michigan for his role as the director of the Michigan Urological Surgery Improvement Collaborative and the Michigan Value Collaborative. Dr Dupree receives salary support from Blue Cross Blue Shield of Michigan for his role as the co-director of the Michigan Value Collaborative and his involvement in the Michigan Urological Surgery Improvement Collaborative. Mr Voit receives salary support from the Michigan Value Collaborative as an account manager and is an employee of ArborMetrix.

 The authors are employed by University of Michigan, which has a contract from Blue Cross Blue Shield of Michigan to operate the Michigan Value Collaborative. 

The Michigan Value Collaborative has created a claims-based algorithm that categorizes claims into episode components. This manuscript describes the validation of this algorithm.

Validation of a Claims-Based Algorithm to Characterize Episodes of Care

 Antibiotic stewardship is key to optimizing patient outcomes and affordable care. The study objective was to examine the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis among providers of varying experience.

A stepped-wedge cluster randomized intervention to evaluate antibiotic use for acute sinusitis encounters at 126 Kaiser Permanente Southern California clinics between September 2014 and April 2015.

 The primary outcome was receipt of an antibiotic prescription. Multivariate analysis adjusted for patient, provider, and medical center characteristics. Secondary analyses described sinusitis and other common upper respiratory infection (URI) diagnoses and antibiotic use during the study period compared with prior years. Chart review of a random sample reported the proportion of encounters receiving guideline-concordant antibiotics. 

Analysis of 21,949 encounters (10,491 pre- and 11,458 post intervention) showed CDS reduced the use of antibiotics (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87), although the pre-post absolute difference was small (85.9% vs 83.9%, respectively). Education had a large initial effect (AOR, 0.51; 95% CI, 0.46-0.57), which did not persist. Increasing years of provider experience raised the rates of antibiotic prescribing, but did not have a significant interaction with CDS (

= .19). The effect of CDS varied by medical center (

<.001). In addition, sinusitis diagnoses decreased post intervention, with no overall increase in antibiotic prescribing for URI diagnoses. Lastly, guideline-concordant antibiotic use increased by 14%. 

Provider education and CDS improved antibiotic stewardship and changed diagnosis patterns. The benefits of education were brief, and CDS effectiveness varied by medical center. 

It is vital to understand the effectiveness of interventions designed to improve patient care in order to disseminate and replicate successful initiatives and avoid fruitless care improvement efforts. Our study of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis offers the following to inform health systems and future research: 

CDS integrated in an electronic health record can discourage inappropriate antibiotic prescribing and will likely change diagnostic patterns. 

Provider education is unlikely to have a sustained effect on antibiotic prescribing.

The effectiveness of education and CDS will vary based on inherent differences among locations.

Antimicrobial stewardship is key to optimizing patient outcomes and minimizing the emergence of drug-resistant bacterial organisms.

 Unfortunately, the goal of reducing inappropriate antibiotic prescribing has remained elusive.

 This is true despite the existence of a variety of strategies to improve antibiotic stewardship, such as pay-for-performance incentives, documentation change strategies, provider education, and social “nudges.”

 We currently do not understand why these approaches have variable results across different settings. Research is needed to describe the effect of such interventions in general community practice and to understand variations in effect due to patient, provider, or medical center characteristics in order to inform current efforts to improve care. 

Acute sinusitis is a condition that affects 31 million annually

 The preponderance of evidence suggests antibiotics are not only ubiquitously overused for acute sinusitis treatment,

 but the correct antibiotic is often not chosen.

 Despite agreement regarding overuse, there is no universal recommendation regarding the precise group of patients with acute sinusitis who may benefit from antibiotics.

 We used the recommendations from our health system, patterned after the Infectious Diseases Society of America (IDSA)’s guideline,

 to report the effects of a 2-component intervention designed to improve antibiotic stewardship for acute sinusitis encounters. The components were computerized clinical decision support (CDS) within an electronic health record (EHR)

Clinical leaders and policy makers need to understand the variable effectiveness of multifaceted interventions, as well as the importance that individual provider characteristics may have on expected results. Our aim was to enhance the understanding of the effects of clinician education and CDS in a real-life clinical setting and, specifically, to assess any differences in effectiveness on providers with varying years of experience or among medical centers. 

We performed a pragmatic stepped-wedge cluster randomized study of provider education and CDS for acute sinusitis encounters. The study was performed in outpatient clinics (105 primary care, 21 urgent care) between September 2014 and April 2015 within Kaiser Permanente Southern California (KPSC), a large integrated health system. This system provides healthcare to more than 4 million members representative of the population diversity found in southern California. We used information from the EHR collected during routine care. This study was approved by the KPSC Institutional Review Board.

Each participating primary care and urgent care clinic is clustered within 1 of 6 medical service areas. Each service area was randomly assigned to receive the CDS intervention as part of a staged implementation over the course of 6 months. The total study period was 8 months, allowing for 1 month of pre-data for all areas prior to CDS implementation. CDS was turned on in 1 of the 6 areas each month until all were receiving CDS, then 1 month of post data was collected after all had received the intervention (

). This staggered implementation allowed for a stepped-wedge analysis.

 This design controls for confounders due to secular trends and strong champion effects. 

CDS was based on qualitative interviews with practicing clinicians

 and designed using features shown to be effective in previous trials.

 Automated in the EHR, CDS was initiated at the time of antibiotic prescribing for patients with an acute sinusitis diagnosis. Once triggered, it offered a clear recommendation based on current evidence.

 The alert informed providers that antibiotics were not generally recommended for patients with symptoms of less than 10 days’ duration. Additionally, CDS included options to justify use of antibiotics in the case of prolonged symptoms, “double worsening” of symptoms, or abnormal physical exam findings. CDS could be bypassed by providers, but required a response or cancellation to proceed with the order. In addition, CDS directed providers to an order set to facilitate the prescribing of amoxicillin or amoxicillin clavulanate if they had ordered a nonrecommended antibiotic for acute sinusitis. 

Provider education included a recorded online presentation and 2 webinars delivered in the middle of the staggered rollout (December). The intervention was created after previous research showed the need for improved stewardship and no temporal trends in improvement in the years leading up to our study period.

International Classification of Diseases, Ninth Revision, Clinical Modification

] code 461.x) taking place in primary care settings were included. We excluded encounters with specialists, follow-up visits, and patients with chronic diseases compromising their immune response (chronic liver disease, 

 codes 140-165, 170-176, 179-209, and 235-239; and common rheumatologic disorders, 

 codes 714, 710, 555.9, and 556). To ensure completeness of data, we limited our sample to Kaiser health plan members. 

The primary outcome was prescription of an antibiotic, classified by the generic product identifier. Secondary outcomes included prescription of an antibiotic for diagnoses with similar symptoms as acute sinusitis (acute nasopharyngitis, 

 code 465; acute upper respiratory tract infections of other multiple sites, 

 code 465.8; acute upper respiratory tract infections not otherwise specified, 

 code 466; influenza with respiratory manifestations, 

 code 487.1; and bronchitis not specified as acute or chronic, 

Concordance estimates were done based on structured chart review from a random sample of 50 charts with some from each of the 6 medical service areas. The structured chart review was performed by a trained member of the research team who followed a standardized protocol to limit bias.

 This random sample was selected from encounters in which antibiotics were prescribed post intervention. We used a standardized data abstraction form and manual previously developed to identify guideline-concordant antibiotic use for similar acute sinusitis encounters, which is proven to have high interrater reliability (93% agreement, kappa = .86).

 This previous research provided estimates of concordance rates (32%) for acute sinusitis encounters receiving antibiotics for comparison purposes. We used results from our chart review and applied the sampling fraction to estimate the variance of the proportion of encounters receiving antibiotics concordant with recommendations post intervention. This estimate was calculated via standard formulas routinely used for survey sampling research,

 then applied to the entire sample to construct 95% confidence limits. 

We used multiple logistic regression analyses to examine the main effect of CDS on antibiotic prescribing. To account for the staggered implementation of CDS, a dichotomous variable was created to indicate whether the encounter was pre- or post CDS. Another indicator variable was included to assess the impact of education amidst the rollout of CDS. A facility variable was included to identify differences in effect among the service areas. The analysis adjusted for variables that were statistically associated (

The final model used the provider as a random intercept, the education indicator as a fixed effect, and indicator variables modeled as fixed effects to assess the separate effects of CDS and provider education. 

Secondary analyses examined the interaction between CDS and provider groups of varying years of experience. We categorized the years of provider experience into 4 groups (<3 years, 3-9 years, 10-20 years, and >20 years), using the largest group of providers (10-20 years) as the reference for comparison. We similarly examined the interaction between CDS and each of the medical service areas. Within these groups of providers and service areas we also compared the odds of receiving antibiotics pre- and post intervention. 

We performed additional analyses to understand changes in sinusitis encounter diagnoses compared with similar upper respiratory infections (URIs) and to evaluate if these encounters resulted in changes to antibiotic prescribing. We report the number of acute sinusitis encounters and other similar URIs, as well as the proportion resulting in antibiotics during the study period (September 2014-April 2015) compared with the same months in the 2 prior years, 2012 to 2013 and 2013 to 2014. Similarly, we report these diagnoses and the percent receiving antibiotics stratified pre-post CDA implementation and pre-post provider education during the study period, as well as the type of antibiotic used pre-post CDS for acute sinusitis encounters. 

The study sample included 21,949 initial acute sinusitis encounters for adults (Figure 1). A similar number of encounters were included before and after the implementation of CDS (10,491 pre- and 11,458 post intervention), and more patients were seen at primary care clinics (76%) than in urgent care settings (

The CDS intervention significantly reduced antibiotic use for acute sinusitis encounters (adjusted odds ratio [AOR], 0.78; 95% CI, 0.71-0.87) (

), although it demonstrated only a 2% absolute reduction. Provider education, assessed as a time effect during December, showed significant decreases in antibiotic prescribing (aOR, 0.51; 95% CI, 0.46-0.57) (Table 2), but these effects were not sustained over the study period (

The odds of receiving antibiotics increased with additional years of provider experience (Table 2), although there was not a statistically significant interaction between years of provider experience and CDS (

). There were significant differences in the effect of CDS and antibiotic prescribing based on the interaction with each medical service area (

The overall number of encounters with acute sinusitis diagnoses decreased compared with the same temporal period in the 2 prior years and represented a smaller proportion of acute URI diagnoses (8.1% vs 11.7% and 11.3%). The proportion of acute sinusitis encounters, compared with all similar URI diagnoses, decreased in the posteducation (7.5% vs 9.3%) and the post-CDS (7.1% vs 9.2%) periods compared with pre-intervention. The overall proportion of acute URI encounters receiving antibiotics decreased during the study period (27.1%) compared with prior years (31.2% and 30.7%), but this does not appear to be attributable to the intervention. Pre-post stratification showed there may have been some diagnosis shifting where antibiotics were still prescribed, as CDS (3.3% increase) and education (2% increase) showed a small increase in antibiotic prescribing for other URIs before and after implementation during the study period (

]). The types of antibiotics used pre-post intervention did not differ significantly (

Lastly, a structured review of acute sinusitis encounters for patients who received antibiotics found that 46% (95% CI, 32%-60%) were guideline concordant. This was a 14% absolute improvement from a structured review of acute sinusitis encounters from the same primary care clinics in the pre-implementation period (32%; 95% CI, 19%-45%).

We performed an effectiveness study of the ability of CDS and provider education to decrease the use of antibiotics for acute sinusitis encounters. Adjusted comparisons showed a 22% improvement in the odds of prescribing antibiotics after the intervention, but the absolute reduction was small (2%). After CDS and education, the number of encounters with an acute sinusitis diagnosis decreased substantially. Lastly, we found that CDS had variable effectiveness based on the medical service area.

This study makes several important contributions to the existing body of literature describing the effectiveness of CDS. First, we confirmed that CDS is effective in real-world clinical settings, but showed that the magnitude of benefit may be less than that observed in highly controlled studies of efficacy.

Second, we observed a pattern of diagnosis-shifting as a result of education and CDS implementation. This may be an example of documented CDS “workarounds,” and future studies and quality-improvement efforts should account for changes in diagnosis patterns.

 It is possible that providers substituted a diagnosis other than acute sinusitis once they were made aware that antibiotics were not indicated for most patients with sinusitis. Understanding how CDS modifies diagnosis patterns warrants future investigation, although, based on our results, these changes did 

 result in an overall increase in the use of antibiotics for all URI diagnoses. 

Third, we observed increasing odds of antibiotic prescribing correlated with increasing years of provider experience, but this did not result in varying CDS impact. CDS did vary in effectiveness, however, based on the medical service area of the encounter. This may be a result of the culture and context within particular settings that promotes success. This finding supports other reports that have shown these less tangible factors to be an important aspect of the success of interventions.

 What is clear is that clinical leaders, policy makers, and researchers should account for the variable effectiveness of CDS depending on the local medical center characteristics. 

Despite overall results demonstrating improvements in antibiotic stewardship and guideline-concordant prescribing, we observed a diminution in effect over time. Much of the overall effectiveness of the 2-component intervention may be attributed to the acute drop in prescribing in December associated with provider education, rather than the CDS intervention. This temporal improvement was not sustained, and by the end of the study period rebounded to near the prior baseline. This reinforces that sustained improvement in clinical practice is difficult to achieve. 

Our study confirms results from recent efficacy trials showing the benefit of “accountable justification”

 with automated CDS during outpatient encounters, although in a slightly different approach. The limited effectiveness of our results may be partially due to limited “accountability,” as our CDS required a clicked response, but did not require the provider to document in the medical record the reason for use of antibiotics. Additionally, we looked only at acute sinusitis encounters, a condition for which antibiotics are generally not indicated, except under special circumstances, instead of viral diagnoses for which antibiotics are always contraindicated. Based on our relatively large aOR and comparatively small absolute effect of CDS, we hypothesize that patients with shorter symptoms or less severe symptoms were those changed to a different diagnosis and not given antibiotics. This is also based on our chart review, which showed improvements in guideline-concordant use of antibiotics. This may explain the increase in other similar URI diagnoses without a rise in antibiotics for those conditions. Therefore, it is reasonable to predict that the effect of our intervention on overall use of antibiotics may be underestimated by the absolute 2% decrease in the proportion of encounters receiving antibiotics for acute sinusitis.

Provider education and integrated CDS reduced antibiotic use for acute sinusitis encounters and substantially changed patterns of diagnosis for acute URIs. The benefits of education were brief and did not persist through the study period, and CDS effectiveness varied by medical center.

The authors thank Ellen J. Rippberger and Lorena Perez-Reynoso for their administrative assistance and organization throughout this study. They also recognize Jason Doctor, PhD, and Daniella Meeker, PhD, for reviewing and providing feedback about the manuscript. 

 Department of Research and Evaluation, Kaiser Permanente Southern California (ALS, YRH, ES, MKG), Pasadena, CA; Department of Emergency Medicine, Los Angeles Medical Center, Kaiser Permanente Southern California (ALS), Los Angeles, CA; Department of Emergency Medicine, San Diego Medical Center, Kaiser Permanente Southern California (RC), San Diego, CA; Department of Emergency Medicine, Kaiser Permanente Northwest (RPR), Portland, OR; Medical Director of Quality and Clinical Analysis, Kaiser Permanente Southern California (MHK), Pasadena, CA.

The authors assessed the effect of provider education and clinical decision support (CDS) on antibiotic prescribing for acute sinusitis. Education and CDS improved antibiotic stewardship and changed diagnosis patterns.

November 2017

Diabetes Care Improvement in Pharmacist- Versus Nurse-Supported Patient-Centered Medical Homes

Validation of a Claims-Based Algorithm to Characterize Episodes of Care

Improving Antibiotic Stewardship: A Stepped-Wedge Cluster Randomized Trial

Electronic Reminder's Role in Promoting Human Papillomavirus Vaccine Use

Changes in Cardiovascular Care Provision After the Affordable Care Act

Reframing the Unaffordability Debate: Patient Responsibility for Physician Care

Impact of Health Reform on Young Adult Prescription Medication Utilization

Patients' Perspectives of Care Management: A Qualitative Study

The Influence of Provider Characteristics and Market Forces on Response to Financial Incentives

Using the 4 Pillars to Increase Vaccination Among High-Risk Adults: Who Benefits?