Use of Patient-Reported Outcomes and Satisfaction for Quality Assessments
Anne P. Ehlers, MD, MPH; Sara Khor, MS; Amy M. Cizik, PhD, MPH; Jean-Christophe A. Leveque, MD; Neal S. Shonnard, MD; Rod J. Oskouian, Jr, MD; David R. Flum, MD, MPH; and Danielle C. Lavallee, PharmD,
Traditionally, clinical outcomes, such as the rates of adverse events or mortality, served as the metrics by which we measured treatment success or failure. Although these objective measures of quality allow for comparisons across hospitals and physicians, the sole use of clinical outcomes may provide an incomplete appraisal of healthcare quality. Within the last few years, payers, policy makers, and healthcare quality organizations focused on patient-reported outcomes (PROs) as important markers of quality distinct from assessments of patient satisfaction and clinical outcomes.1-4 PROs are defined as measures of health that are directly reported by the patient, rather than interpreted by a clinician or reported in the medical record.5 This newer metric is becoming financially relevant as a result of the Merit-Based Incentive Payment System (MIPS), passed as part of the Medicare Access and CHIP Reauthorization Act of 2015.6 PROs have been proposed by policy makers and payers as a required component of quality assessments and reimbursement schedules.7,8 As clinicians and hospitals are charged with adding PRO measures to established clinical and satisfaction metrics, a challenge exists to integrate PROs in a manner that both improves patient care and provides a meaningful assessment of quality.
Many PRO scores assess a patient’s perception of improvement after an intervention, but others aim to quantify a patient’s level of satisfaction with their care. However, it is not clear that these patient satisfaction measures correlate with the delivery of safe and effective care as defined by clinicians and caregivers. Satisfaction surveys are often criticized for focusing on the “wrong” outcomes, such as the hospital noise level or the variety of food choices offered to patients, rather than on the clinical result of a treatment or intervention.9 Patient satisfaction also may not reflect the total patient experience, as one patient might report high levels of satisfaction with their overall care despite a poor clinical outcome10 and another patient might report dissatisfaction with their care while simultaneously benefiting from a positive clinical outcome.
Spine surgery represents 1 relevant scenario in which integration of both PRO and clinical outcome measures improves the holistic view of an individual’s outcome following surgery. Functional impairment and pain are often the primary drivers for patients considering spine surgery; therefore, understanding which patients are most likely to see improvement in these symptoms can inform the decision to be made between a surgical and a nonsurgical option. Using PROs to inform direct patient care versus measures for assessing changes in a patient population is increasingly called for in today’s healthcare environment.11,12 In Washington state, healthcare providers are experiencing demands to implement and interpret PROs in spine surgery due to recent policy recommendations from the Bree Collaborative, proposed in 2015. The Bree Collaborative calls for the use of the Oswestry Disability Index (ODI) and the Patient-Reported Outcomes Measurement Information System (PROMIS) 10, measures of functional and general quality of life, respectively, for all patients undergoing single-level lumbar spine fusion procedures. This policy is intended to allow appropriate selection of surgical candidates from the vast population of adults with low back pain and to limit expensive surgical interventions to those patients most likely to benefit from the procedure.13
Given the differences between PROs, which typically focus on functional ability and pain, and patient satisfaction measures, it is not clear that one or the other necessarily provides a true measure of outcome or that these measures are robust enough to allow for policy and reimbursement decisions. To address this, we investigated the relationship between PRO measures and patient satisfaction at 1 year following spine surgery. We hypothesized that there would be significant disparities between patient satisfaction and PRO measures, suggesting that the use of satisfaction scores as a primary measure of quality is flawed.
The Comparative Effectiveness Research Translation Network (CERTAIN) is a research and analytic platform aligned with the Spine Surgical Care and Outcomes Assessment Program (Spine SCOAP).11,14,15 Spine SCOAP is a collaborative of hospitals within Washington state whose mission is to aggregate data from hospitals, healthcare providers, and patients to generate evidence-based best-practice measures that can be dispersed among the collaborative to improve the care of patients in the state and beyond. We conducted a retrospective cohort study of adults undergoing elective lumbar spine surgery from 2012 to 2014 at 12 Spine SCOAP hospitals who also participated in CERTAIN PRO data collection.
Eligible participants completed a survey that included measures of pain, function, and satisfaction both before surgery and at 1 year after. Pain was measured using a 10-point numeric rating scale with “0” representing “no pain” and “10” representing “worst pain possible.” Function was measured using the ODI, a composite measure that grades function on a scale of 0 to 100, with higher scores indicating worse disability.16 Improvement in pain or function was established by a minimally clinically important difference between baseline and follow-up (for pain, a change of at least 2 points from baseline; for function, a change of at least 15 points from baseline).17 We measured satisfaction using a single question: “How satisfied are you with the overall result of your spine operation so far?” The satisfaction question was scored using a 4-point modified Likert scale (Table 1).
The University of Washington Human Subjects Division approved the retrospective review of these data. All analyses were carried out using Stata version 11 (STATA Corp; College Station, Texas).
We analyzed patient-reported satisfaction, pain, and function at 1 year post operation to determine the association between patient characteristics and a positive outcome in any or all domains. Patient characteristics were obtained through Spine SCOAP and included information about sociodemographics, underlying diagnosis (eg, spinal stenosis, herniated disc), and operative characteristics (eg, fusion). An invasiveness index based on the type of intervention at each vertebral level as well as the number of operated levels was included, as this has been shown to be correlated with clinical outcomes in spine surgery patients.18,19 The American Society of Anesthesiologists Physical Classification System score was included as an additional measure of physical status at the time of the operation. This is a commonly used score applied to every patient undergoing surgery and is intended to provide an additional description of patient risk.20 We selected the 1-year postoperative time point because PROs and satisfaction in the early postoperative period are not necessarily reflective of findings at 1 year,21,22 which are felt to be a more reliable indicator of long-term satisfaction.
To determine the relationship among the 3 domains, we used a Poisson regression model stratified by pain and function improvement that was predictive of satisfaction (somewhat or very satisfied) at 12 months following spine surgery. A subset of patients (n = 69; 13%) did not have clinical information available through Spine SCOAP and was excluded from the regression analysis. To account for the fact that other patient characteristics are known to correlate with overall satisfaction,23 we included sociodemographic factors, diagnostic factors, and operative characteristics as measured by SCOAP in the regression model. The resulting model was clustered by hospital to account for correlated data. In cases such as this, where multiple observations are drawn from the same hospital, there is a concern that these results from within each individual hospital are correlated due to extrinsic factors that are not measured and that this effect varies between hospitals. Failure to account for correlation can result in a biased estimate and an incorrect inference.
At the 1-year follow-up point, 520 patients (mean age = 63 ± 13 years; 47% male) had complete baseline and follow-up survey data and 451 (87%) had clinical information available from Spine SCOAP. More than half of patients reported that they were very satisfied (n = 301; 58%) while one-fourth of patients reported that they were somewhat satisfied (n = 124; 24%) with their spine surgery result. Table 2 demonstrates the interaction among patient-reported pain and function improvement and patient satisfaction. Patients with a higher magnitude of pain improvement more often reported that they were somewhat or very satisfied: 62% patients with no pain improvement or worse pain were satisfied, whereas 96% of those with 5 or more points of pain improvement were satisfied. We found a similar trend in satisfaction and function: 61% of those with no functional improvement or worse function reported being satisfied and 80% of those whose function improved more than 30 points reported being satisfied. Interestingly, even among patients who had no change in their symptoms or reported that they were worse, 59% of patients were still somewhat or very satisfied.
In our stratified analysis, adjusting for patient characteristics, patients who had no improvement in back pain were 1.7 times (95% CI, 1.27-2.16) and 2.3 times (95% CI, 1.71-3.11) more likely to be satisfied if they experienced a minor improvement in function (ODI score reduced 15-29 points) or major improvement in function (ODI score reduced ≥30 points), respectively, compared with patients who had no improvement or worse function. Of the patients with improved back pain at 1 year, a mild improvement in function was associated with 1.3 times (95% CI, 1.00-1.71) the likelihood of being satisfied, and a major improvement in function was associated with 1.4 times (95% CI, 1.08-1.81) the likelihood of being satisfied compared with patients who had no improvement or worse function outcomes (Table 3).
In this analysis, we sought to determine how PRO measures compare with other quality measures, such as patient-reported satisfaction. We analyzed data from a statewide clinical registry encompassing a collaborative of hospitals in Washington state that allowed for an assessment of both hospital-level and patient-level outcomes. The majority of patients reported being satisfied with their spine surgery result, and we found a strong association between patient-reported improvements in pain and function and satisfaction. Despite this understandable finding, more than half of the patients with no improvement or worse outcomes in pain or function were satisfied with their surgery. This incongruence suggests that satisfaction ratings may be based on nonclinical aspects of care not captured by this survey. Reasons for this incongruence may be related to the patient’s perception that their care was especially respectful or well coordinated. Conversely, there was a group of patients who did improve but were not satisfied, which may indicate that they had other expectations beyond pain or functional improvement. We can conclude that the “quality” profile of a hospital is highly dependent on the domain of measurement and that understanding the items measured by a particular instrument is critical for interpretation of results.
Previous studies have explored the relationship between patient satisfaction and other outcome measures (eg, clinical outcomes, patient-reported functional outcomes), but conclusions have varied among studies. In one prospective study, Godil et al reported that patient satisfaction measures at 90 days post operation were not correlated with clinical outcomes, such as complications and readmissions, and as such, they should not be used as a proxy for overall quality.24 These findings were confirmed in a subsequent review article.8 The time frame of measurement is important, as well: PROs 3 months post operation are not necessarily predictive of PROs at 1 year.21,22 This latter finding may be clinically relevant as satisfaction metrics are frequently measured at time points close to the date of care delivery, rather than at later time points, which might better reflect long-term improvements or outcomes. These studies highlight the unresolved issues related to the use of PROs and satisfaction data in quality assessments.
More generally, there is evidence that satisfied patients are more likely to utilize healthcare resources and thereby incur costs, but it is not clear that the increased healthcare utilization is related to better outcomes.25 Among surgical patients, one study found no association between satisfaction and measures of quality as defined by the Surgical Care Improvement Program, such as adequate antibiotic prophylaxis before surgery,26 again pointing to the concern that patients base satisfaction ratings on nonclinical factors that are not reflected in clinical or administrative data. Our analysis found a correlation between patient-reported improvement in pain and function and satisfaction in the patient-level analysis, but there was a relatively large proportion of individuals who reported being somewhat or very satisfied who also had a negative outcome in either pain or function. Potential reasons for this include appreciation of the nursing care that the patient received, the organization of their care, or attentiveness of their care team as a whole. Future work should focus on understanding the domains of care that are most correlated with satisfaction.
The survey utilized in this study was created under the auspices of quality improvement to help surgeons and hospitals understand more about their patients. The evidence is still emerging as to the complete set of questions that are relevant to support patient- and hospital-level decision making, as well as to provide quality assessments. One limitation of this analysis is the potential inability to capture all domains that are important to patients, and thus it may be biased to those areas deemed clinically relevant to the survey creators. A second limitation is the restriction of the analysis to lumbar surgery patients alone, meaning the results may not be reflective of all clinical conditions. We encourage future research endeavors that focus on how measures of the patient experience correlate in other health conditions. Because this analysis included only those with complete data, these results may not be representative of outcomes from nonresponders. We do not have information regarding nonresponders, and it may be that patients who respond to surveys are systematically different from those who do not. It may also be that improvement in pain or function is representative of better patient selection. Although this is outside the scope of this research, future work should focus on understanding which patients will benefit most from spine surgery. Finally, the survey supported through CERTAIN is currently only available in English, which may further bias our results.
In Washington state, current policy and payment recommendations require health systems to capture PRO data for patients undergoing lumbar spine surgery. It is imperative that we understand how to interpret this PRO information to ensure that patients get the right care at the right time. With the ongoing concerns about PRO implementation, collection, interpretation, and application,27,28 it is necessary to focus on how these important data can inform decisions about patient care. Successful implementation depends not only on the available infrastructure for collection and analysis, but also on the commitment of healthcare providers to the concept that PROs add value to the ongoing quality discussion. Without this focus, we risk more measurement and more data without the ability to transform patient care. There is considerable work to be done before we fully understand the appropriate role of PROs in healthcare.
The implications of this work are far-reaching, especially with the recent passage of MIPS, which emphasizes quality and value over volume.6 Results such as this may serve to inform policy makers who are deciding on the most appropriate measures to use in quality assessments. Understanding the relationships among PROs, satisfaction, and general quality is the first step in drawing meaningful conclusions that can then be translated to policy. From a patient’s perspective, our study is important because it may help patients better interpret the significance of a particular physician or hospital having high satisfaction ratings. Furthermore, this work requires the inclusion of patients, providers, payers, and policy makers to ensure that we are selecting the appropriate domains on which to judge quality.