Validation of a Claims-Based Algorithm to Characterize Episodes of Care
Chad Ellimoottil, MD, MS; John D. Syrjamaki, MPH; Benedict Voit, MBA; Vinay Guduguntla, BS; David C. Miller, MD, MPH; and James M. Dupree, MD, MPH
Hospitals are increasingly being held accountable for services and expenditures that occur beyond the hospitalization through episode-based performance measures.1-6 Postdischarge expenditures, such as postacute care and readmissions, have been cited as the fastest growing spending categories over the last 2 decades and have been the target of many national programs focused on reducing healthcare costs.3 For example, CMS recently implemented the Comprehensive Care for Joint Replacement bundled payment program, which will hold hospitals financially accountable for expenditures occurring from admission through 90 days post discharge.7 In addition, accountable care organizations were developed to reduce costs that occur both outside of and during hospitalizations.8 Based on the prevalence and growth of episode-based payment programs, it is evident that many payers believe the key to reducing healthcare expenditures is to hold hospitals responsible for efficiency along the entire patient care episode.
Despite enthusiasm for increased episode efficiency, identifying specific high-cost events, such as readmissions, can be challenging for hospitals for several reasons.First, it is difficult for hospitals to track events outside of the initial hospitalization. In fact, hospitals are often not even aware of postdischarge events that occur at outside facilities. Secondly, unless directly affiliated with the hospital, postdischarge providers are not incentivized to report utilization patterns to hospitals. Furthermore, many small hospitals may not have internal resources to monitor and track postdischarge events and spending.
In Michigan, one response to these challenges was the development of the Michigan Value Collaborative (MVC).9 Established in 2012 and funded by Blue Cross Blue Shield of Michigan (BCBSM), MVC’s mission is to provide hospitals with episode-level data and promote high-quality care at the lowest reasonable costs.10 One particular area of interest has been postacute care, specifically rehabilitation. MVC hospitals have begun to use episode-level data to monitor rehabilitation expenditures, especially for conditions such as acute myocardial infarction (AMI) and hip replacement, where more than one-third of all patients are discharged to rehabilitation facilities (Table 1).However, a necessary step in this process is to ensure that MVC’s claims-based identification of postdischarge services is accurate.
In this context, we describe a large-scale medical records–based validation of the algorithm used by MVC to define clinical episodes of care in commercial claims. We believe that the MVC validation experience will provide useful insight to hospitals and payers about the advantages and limitations of using claims to track events that occur after hospitalization.
The MVC collects claims data for BCBSM beneficiaries admitted to 1 of 63 Michigan hospitals for 21 medical and surgical conditions. We consulted clinical experts who used International Classification of Diseases, Ninth Revision (ICD-9) diagnosis/procedure and Current Procedural Terminology (CPT) codes to define the conditions. Then, we utilized variables such as revenue codes, diagnosis-related groups (DRGs), facility identification, and CPT codes to classify individual claims into: inpatient, skilled nursing facility (SNF), home health, emergency department (ED), inpatient rehabilitation, outpatient rehabilitation, and general outpatient claims. Cases with readmissions to hospitals other than the index facility were excluded from the readmissions validation step. eAppendix A (eAppendices available at ajmc.com) further details the entire attribution process. Table 1 presents the characteristics of episodes identified from claims data by condition type, including use of postdischarge services. This study was deemed exempt from review by the Michigan Institutional Review Board.
Our validation process occurred in 2 phases (pilot and full validation). During the pilot, 6 hospitals were asked to review 10 to 20 BCBSM preferred provider organization (PPO) cases from between January 1, 2013, and October 31, 2014. For matching purposes, the clinical condition, national provider identifier, date of birth, gender, admission date, and discharge date for each patient was provided. Specifically, participants were instructed to indicate if the patient listed had records demonstrating that a specified event occurred within 90 days of their discharge date.These events included 30-day readmissions, ED visits, SNF admissions, home health visits, and rehabilitation services (inpatient and outpatient).
The lessons learned from the pilot were used to inform the full validation. Here, we distributed the same key variables to all 63 MVC participants. Each hospital was provided 30 cases to review, with the exception of hospitals that participated in the pilot (which were asked to review 20 cases). We selected conditions based on volume, prevalence of associated postdischarge services, and suggestions from clinical experts. The conditions included colectomy, coronary artery bypass graft, AMI, pneumonia, congestive heart failure, hip replacement, knee replacement, cesarean delivery, vaginal delivery, trauma, and spine surgery. In total, 1830 BCBSM PPO cases were selected for data validation from these 11 conditions.
We identified areas of agreement and disagreement between MVC’s claims-based algorithm and medical records. First, we looked for agreement that these episodes occurred and were attributed to the correct hospital and to the correct condition. Next, we used a kappa statistic to assess agreement for postdischarge services (eAppendix B). For each disagreement in a case, 2 members of the MVC team, a clinician and an analyst, reviewed the specific claims. After determining the cause of the discordance, we adjusted our algorithm to better capture hospital events and re-evaluated the level of agreement.
In the cases where MVC reported a postdischarge event that did not match a hospital’s medical records, we used confirmatory evidence from the claims to improve our confidence that the event occurred. Specifically, we examined the following items: 1) Did the postdischarge event have more than 2 claim line items with dates following the index admission?; 2) Did the place of service designation on the claims support the assignment of the claim to the postdischarge service?; and 3) Did the revenue center code on the claims support the assignment of the claim to the postdischarge service?
One hundred percent of hospitals (n = 63) participated in the validation process. Hospitals matched 1812 of the 1830 (99%) MVC episodes to records in the hospital’s medical charts.
The agreement for the occurrence of postdischarge services ranged from 0.16 to 0.78, with rehabilitation services having the poorest agreement. eAppendix B contains details of the validation process.
Using administrative claims, we were able to identify 183 readmissions for 1812 episodes (10% readmission rate). Of these, there were 15 cases in which MVC observed a 30-day readmission that was not evident in the hospital data (Table 2).These were due to readmissions to a hospital different than the index facility and, per the methods section, were excluded. There were 24 discordant readmissions which MVC did not identify and a hospital did; these cases were classified as observation unit stays in MVC data (eAppendix B).
We identified 452 ED visits (25% of episodes). Of these visits, 292 (65%) were identified by hospitals. Importantly, there were 160 out of 452 cases (35%) in which MVC observed an ED visit that was not reported in the hospital clinical data (Table 2). Of the 160 discordant cases, 122 (76%) of these ED visits occurred at a different facility than the index admission. All 160 cases had confirmatory evidence of a valid ED visit.
There were 85 episodes (5%) in which a hospital reported an ED visit that was not evident in the MVC data. Upon review, many of these were classified elsewhere. For example, 38 preceded an index admission or a readmission and the ED services are grouped with this hospitalization in the MVC episode (eAppendix B).
We identified 64 SNF admissions (4% of episodes). Of these, 43 (67%) admissions were identified by hospitals. There were 21 cases (33%) in which MVC identified a SNF admission but the admissions were not reported by hospitals (Table 2). We found evidence in the claims data to confirm that all 21 visits occurred. There were only 9 cases in which hospitals reported care in a SNF, and we found no evidence of a SNF admission in the MVC claims data (eAppendix B).
Home Health Visits
We identified 949 home health visits (52% of episodes). Of these visits, 781 (82%) were identified by hospitals (Table 2). Of those reported by hospitals, there were 168 cases (21%) in which MVC observed a home health visit that was not evident in the hospital clinical data. There were 99 cases (10% of episodes) in which a hospital observed a home health visit that was not evident in the MVC data (eAppendix B).
The algorithm identified 1223 rehabilitation visits (67% of episodes). Of these visits, 350 (29%) were reported by hospitals. There were 873 cases (71%) in which MVC observed a rehab visit that was not evident in the hospital clinical data (Table 2). Of the 873 discordant cases, 851 (97%) had confirmatory evidence in the MVC claims for utilization of rehabilitation services. There were only 42 cases (3% of episodes) in which a hospital observed a rehab visit that was not evident in the MVC data (eAppendix B).
Improvements to the Claims-Based Algorithm
After reviewing all cases of discordance between the claims-based algorithm and the medical records, we identified several areas for improvement, the most important being to update the ICD-9 codes used to identify postdischarge claims to better reflect the coding practices used by hospitals. For example, ICD-9 code V57.8 (care involving other specified rehabilitation procedure) was initially not considered by MVC as a related diagnosis code for patients after joint replacement. However, this code was used for a large number of rehabilitation services after joint replacement. After making this and other improvements, we re-evaluated the level of agreement between our algorithm and the medical records and found improvement in agreement for all services (eAppendix B).
In this study, we validated the MVC’s claims-based algorithm for the identification and classification of postdischarge events. During the process, we found a high episode match rate between MVC data and medical records. Much of the disagreement was due to the inability of hospitals to identify readmissions, rehabilitation services, and other postdischarge events. Collectively, these findings suggest that this claims-based algorithm outperforms medical records in identifying postdischarge events.
Previous investigators have convincingly demonstrated that variation in episode spending is largely due to postdischarge events.3,11-16 Preventable readmissions and variations in postacute care could indicate areas to improve hospital efficiency and outcomes. Others have also validated and demonstrated on a health-system level the success of claims-based algorithms in identifying hospital events.17-19 Existing literature has demonstrated the effectiveness of utilizing these tools to identify high-cost inpatient events and improve the value of care.20 It is reasonable to believe that by using this same approach to identify postdischarge events, institutions could potentially achieve similar results outside the hospital setting. The current study findings demonstrate, on a large statewide scale, that accurately identifying and measuring postdischarge utilization may be difficult for hospitals using medical records alone.A claims-based algorithm could better identify postdischarge events, especially those that occur outside hospitals’ networks. With the current national focus on episode efficiency, identification of these events is imperative to driving high-value care.
Our study has several limitations. First, we only included BCBSM patients in our validation process. Due to CMS privacy restrictions, we were unable to validate our algorithm with Medicare beneficiaries. Second, the MVC algorithm may not be generalizable for other commercial payers, although BCBSM is the largest commercial payer in Michigan. Third, we did not have hospitals look for readmissions that occurred outside of the hospital where the index event occurred. This decision was primarily made to reduce the chart review burden to hospitals; we received early feedback that hospitals could not identify these particular events. Finally, we did not validate our classification algorithm for other postdischarge events (eg, outpatient procedures) and intensity of services (eg, SNF length of stay). However, this study was focused on validating the occurrence of major postdischarge services. Although we did not end with perfect agreement between the MVC data and the medical records, there were only a few events identified by hospitals not seen in MVC claims (0.5%-4%).
Our findings will help stakeholders understand the opportunities and challenges of using a claims-based algorithm to measure episode spending. Relevant to hospital administrators, the finding that the claims-based algorithm used in this study outperformed medical records suggests that such data provide more complete intelligence about the postdischarge period. This finding should encourage hospital administrators to obtain additional claims data by participating in a statewide, regional, or health-system collaboration and by asking payers to share these data. Without these claims data, hospitals will be limited in their ability to measure and optimize services provided outside of their facilities. This is particularly important as CMS and commercial payers are increasingly using episode-based performance measurement and payment bundling.
Moving forward, research in this area should focus on how these data can be refined to provide more granular information to hospitals. For instance, providing hospitals with data on the average length of stay and intensity of services provided at SNFs may help providers understand the efficiency of facilities where patients are sent after discharge. Ultimately, the value of episode-based performance measurement and bundled payment programs as mechanisms to drive high-value care will strongly depend on the accurate measurement of episode-level payments and utilization.
The authors thank Dr Vinita Bahl (University of Michigan), Kelly Rice (Beaumont Health System), Steve Lewis (St. Joseph Mercy Health System), Meghan Coughlin (MidMichigan Medical Center), and John Robertson (Hillsdale Community Health Center) for their comments and insight during the course of this research. They would also like to thank Dr David Share, Ellen Ward, Tom Leyden, and the Value Partnerships at BCBSM for their ongoing support of the Michigan Value Collaborative and this manuscript.