Currently Viewing:
The American Journal of Accountable Care December 2015
The Need to Level the Playing Field Between Accountable Care Organizations and Medicare Advantage
David Introcaso, PhD
The Debate Over Drug Costs: Instead of "How Much" We Spend, Let's Focus on What We Get in Terms of Health
A. Mark Fendrick, MD Co-Editor-in-Chief, The American Journal of Managed Care Professor of Medicine, School of Medicine Professor of Health Management and Policy, School of Public Health Director, Cen
The Pediatric ACO: A New Frontier in Accountable Care
Anthony D. Slonim, MD, DrPH
Revisiting the Role of Academic Medical Centers in Medicare Shared Savings Program ACOs
Benjamin M. Gerber, JD
The Emerging Business Models and Value Proposition of Mobile Health Clinics
Khin-Kyemon Aung AB; Caterina Hill, MSC, MA; Jennifer Bennet, BS; Zirui Song, MD, PhD; and Nancy E. Oriol, MD
Now That ACOs Are Engaged, What About Those Patients?
Mary K. Caffrey
Currently Reading
It Is Time to Ask Patients What Outcomes Are Important to Them
Leif I. Solberg, MD; Stephen E. Asche, MA; John C. Butler, MD; David Carrell, PhD; Christine K. Norton, MA; Jeffrey G. Jarvik, MD, MPH; Rebecca Smith-Bindman, MD; Juliana O. Tillema, MPA; Robin R. Whi
Bundled Payments for Diabetes Care and Healthcare Costs Growth: A 2-Year Follow-up Study
Sigrid Mohnen, PhD; Caroline Baan, PhD; and Jeroen Struijs, PhD
Using Guidelines: When Is It Appropriate?
Maude St-Onge, MD, PHD, FRCPC
The Impact of Direct Oral Anticoagulants in the Managed Care Environment
Charles E. Mahan, PharmD, RPh, PhC; and Robert Lavender, MD, FACP

It Is Time to Ask Patients What Outcomes Are Important to Them

Leif I. Solberg, MD; Stephen E. Asche, MA; John C. Butler, MD; David Carrell, PhD; Christine K. Norton, MA; Jeffrey G. Jarvik, MD, MPH; Rebecca Smith-Bindman, MD; Juliana O. Tillema, MPA; Robin R. Whi
Patients with abdominal or back pain identified 21 outcomes important to them, but the reported outcomes are quite different from the symptom and function outcomes studied by researchers.
Figures 2 and 3 provide graphic comparisons between patient and physician importance ratings for each of the outcomes, or­dered by the magnitude of the difference between them. For ab­dominal pain, physicians rated the importance to patients to be an average of 0.6 points lower than the ratings of patients (3.53 vs 4.09; P = .04); for back pain, that difference was 0.85 points lower (3.65 vs 4.50; P = .0003). Each of the 4 individual out­comes with statistically significantly lower physician ratings for abdominal pain was also significantly lower for back pain. There were also another 9 of the outcomes for back pain with statis­tically significant physician-patient differences. Moreover, all of the items with P <.01 or less were in the back pain list. The out­comes with greatest agreement between patients and physicians were finding the cause, getting rapid and complete relief, and re­turning to work and productivity as soon as possible. For most outcomes, the 3 younger physicians provided lower ratings than the 8 with longer practice experience. Patient ratings of abdomi­nal pain outcomes tended to show more variability than physician ratings, with 15 of the 21 SDs for abdominal pain outcomes be­ing higher for patients than physicians. However, for back pain, the pattern was reversed, with 7 of the 21 SDs for back pain outcomes being higher for patients than physicians.

When the physicians were asked about the most important outcomes from their perspective, 5 highlighted making a diagno­sis, 6 thought it was needed to guide treatment, and 4 wanted to rule out something serious. Four physicians also noted a reduc­tion in symptoms and 3 identified restoration of normal function as soon as possible. Single physicians mentioned humane care, prognosis, and good access to care.


These patients and their families rated a wide variety of out­comes from their care as important. Although there were differ­ences between the average importance ratings by type of pain, the differences were mostly small and all of the outcomes we asked about had relatively high ratings that were similar between patients and family members. Finally, the primary care physician interviewees tended to rate each of these outcomes as having lower importance than did patients or their families, even though the physicians were asked to rate the importance of each out­come from the patients’ perspective rather than from their own. The similar shape of the importance difference curves in Figures 2 and 3 across all of these outcomes suggests that these dif­ferences are real. Such differences highlight the importance of asking patients—both individually and collectively—about what outcomes are important to them.

The results of this study raise important questions about what approach should be used to identify PROMs. The current fo­cus on using measures that have been identified by researchers and those limited to specific functions and symptoms will likely not capture the same outcomes that patients themselves might identify as important. McClimans, in his study, has objected to the whole idea of standardized questions, instead proposing a theoretical framework for thinking about PROMs that under­stands them “as posing genuine questions to patients—questions that are open to reinterpretation [and context].”4 He particularly objects to measures that break down the outcome into a series of sub-questions and that assume their sum fully captures the overall outcome. McKenna raises a similar concern in his study, while emphasizing the importance of a hard science approach to measure development.5

Only one of the outcomes that these patients thought were im­portant (ie, to get rapid and complete relief from pain and other symptoms) would be partially measurable by the PROMIS mea­surement sets established by the National Institutes of Health to measure patient outcomes.6 PROMIS has developed question­naires to ask about pain, fatigue, gastrointestinal symptoms, arm and leg function, sexual function, sleep, physical activity, affect, cognition, self-efficacy, substance use, social support, peer and family relationships, and social roles and activities. There are also global health status questions.7.8 The individual question sets (short forms) for each of these domains varies from 4 to 10 questions selected from banked sets that contain as many as 121 questions. Their psychometric qualities are mostly well stud­ied, and they are a wonderful resource for researchers, but there has been no patient involvement in selecting or reviewing these measures.3,7,8

The other main source for patient outcome measures is the National Quality Forum. It has endorsed nearly 700 measures created by other organizations, 208 of which are identified as outcome measures.9 Although most of the latter are clinical out­comes related to specific conditions, some do address patient function, usually in relation to a specific procedure—few are as generic as the outcomes considered important by our patient subjects. A similar resource in Britain, the Health and Social Care Information Centre, began work in 2009 on PROMs and current­ly has measurement sets in place for hip and knee replacement, hernia repair, and varicose vein treatment—again, mostly clinical and specific for a procedure and condition.10 There is no indica­tion on its website that patients had any role in identifying these measures.

PCORI has prioritized developing patient-centered outcome measures and is becoming a resource for PROMs. It is devel­oping standards for such measures, but so far has not created or endorsed any specific measures. Its Methodology Committee has recommended that patients be included in the peer review process for grant proposals, and it has a standard requiring that researchers “engage people representing the population of inter­est and other relevant stakeholders in ways that are appropriate and necessary in a given research context.”11

Developing outcome measures that are meaningful to patients is a new endeavor for healthcare, and accordingly, it will take some time to get it right. Once we know what patients want mea­sured, learning where to most efficiently and accurately obtain that information is another challenge. Although it might seem obvious that the best source of data is directly from patients, this may not always be feasible because of the cost and difficulty tracking and obtaining responses. Thus, we need to learn whether some of those outcomes can be obtained from medical records or insurance claims data. Yet another challenge will be learn­ing how to incorporate those measures into the daily practice of medicine. Black suggests in his study that outcome measures could be transformative for healthcare, but admits that their use in routine practice is still uncommon, and identifies a series of challenges.12 As if to demonstrate that, his group recently pub­lished a study using PROMs to compare surgeons and conclud­ed that the choice of outcome measure can substantially alter a surgeon’s rating.13 Valderas et al performed a systematic review of the literature on the impact of measuring PROMs in clinical practice.14 They confirmed that “contexts and interventions that will yield important benefits remain to be clearly defined.”


Our findings in this study are limited to patients with 2 types of clinical situations: back pain that tends to be chronic or recur­rent and abdominal pain that tends to be acute. The underlying disease processes and patient impacts of these 2 situations are both heterogeneous and different, which highlights how import­ant it is to survey patients across a broad range of contexts if the goal is to fully understand patient preferences. The advantage of surveying patients as we did was that having their own recent experience with these potentially serious and disabling problems should have made their responses less abstract. All of the sample sizes in this study were small, and the subjects were not especially diverse or representative nationally, so these findings require con­firmation in larger more representative, diverse samples. It is also likely that we have not identified all of the important outcomes.

Nevertheless, the findings are especially important in suggest­ing that there are a wide variety of outcomes that are important to patients and their families, outcomes that are not necessarily limited to relief of particular symptoms or recovery of certain functions. Thus far, most of the attention to PROMs has focused on the psychometric and feasibility aspects of measures that are limited to specific symptoms and functions. It might be wise to first obtain more input from patients before once again learning that the outcomes we measure so well have little relevance to what patients want to know before they make decisions about their care.


The authors are grateful to the skilled interviewers who conduct­ed all of these interviews so well, and to Cally Vinz, vice president at Institute for Clinical Systems Improvement, who has been an active participant in our meetings and helped identify and recruit physicians for interviews. The authors are also extremely grate­ful to the patients, family members, and physicians who have provided them with so much helpful input; this includes Aimee Vang, Carol Dexter, Katharine Pittman, Jamie Gudknecht, Jen­ny Zakoski, David Freedland, and Gail Soens from the Patient/ Family Advisory Board, who have been provided with a modest honorarium for their time. As advisers, they have each provided very helpful feedback on our survey protocol and pilot tested and provided suggestions to improve the survey described in this paper. In addition, John Wilkinson, MD; Ian Kenning, MD; and Deepti Pandita, MD, from the Physician Advisory Board have reviewed the authors’ plans and findings and provided practicing physician perspectives that have helped us to remain focused on that part of their targeted audience, even though we were unable to provide them with any recompense.

The funding agency, PCORI, had no role in the design and conduct of the study; collection, management, analysis, and in­terpretation of the data; and preparation, review, or approval of the manuscript project.

Author Affiliations: HealthPartners Institute for Education and Research (LIS, SEA, JOT, RRW, JYZ), Minneapolis, MN; Health­Partners Medical Group (JB), Minneapolis, MN; Group Health Research Institute (DC), Seattle, WA; University of Washington (JGJ), Seattle, WA; University of California (RS-B), San Francis­co, CA; Saint Paul, MN (CN).

Funding Source: This work was supported through a Patient-Cen­tered Outcomes Research Institute (PCORI) Pilot Project Pro­gram Award. All statements in this report, including its findings and conclusions, are solely those of the authors and do not nec­essarily represent the views of PCORI, its Board of Governors, or Methodology Committee.

Author Disclosures: Dr Solberg is a member of the ICSI regional board (Q1 collaborative), an employee of HealthPartners health system, and has past grants received from CMS, National Insti­tute of Health, AHRQ, and PCORI. Dr Jarvik is a member of the General Electric-Association of University Radiologists Re­search Academy Fellowship Advisory Board, has received a grant from PhysioSonics, an ultrasound-based diagnostic company, and also receives royalties from PhysioSonics. Dr Smith-Bindman has previously received a grant from PCORI for an unrelated study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (LIS, CKN, RS-B, JB, RRW); acquisition of data (LIS, DSC, JB, JOT, RRW, JYZ); analysis and interpretation of data (LIS, SEA, DSC, CKN, JGJ, RS-B, JB, JOT, RRW, JYZ); drafting of the manuscript (LIS, SEA, RS-B, JB); critical revision of the manuscript for important intel­lectual content (LIS, SEA, DSC, CKN, JGJ, RS-B, JB, JOT, RRW, JYZ); statistical analysis (SEA, RRW); obtaining funding (LIS, RS-B); supervision (JOT); and providing content expertise on back pain (JGJ).

Address correspondence to: Leif I. Solberg, MD, HealthPartners In­stitute for Education and Research, PO Box 1524, MS#21111R, Minneapolis, MN 55440-1524. E-mail leif.i.solberg@healthpart­

1. Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol. 2013;66(suppl 8):S12-S20.

2. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Insti­tute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583-1584.

3. Cella D, Riley W, Stone A, et al; PROMIS Cooperative Group. The Patient-Re­ported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol. 2010;63(11):1179-1194.

4. McClimans L. A theoretical framework for patient-reported outcome measures. Theor Med Bioeth. 2010;31(3):225-240.

5. McKenna SP. Measuring patient-reported outcomes: moving beyond misplaced common sense to hard science. BMC Med. 2011;9:86.

6. Instruments available for use in Assessment Center. Assessment Center website. Updated November 10, 2015. Accessed November 2015.

7. Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measure­ment information system (PROMIS) global items. Qual Life Res. 2009;18(7):873-880.

8. Revicki DA, Kawata AK, Harnam N, Chen WH, Hays RD, Cella D. Predicting EuroQol (EQ-5D) scores from the patient-reported outcomes measurement infor­mation system (PROMIS) global items and domain item banks in a United States sample. Qual Life Res. 2009;18(6):783-791.

9. Measures, reports & tools. National Quality Forum website. http://www.quality­ Accessed November 17, 2015.

10. Finalised patient outcome measures (PROMs) in England: April 2011 to March 2012. Health & Social Care Information Centre website. catalogue/PUB11359/final-proms-eng-apr11-mar12-fin-report-v2.pdf. Published January 7, 2014. Accessed November 2015.

11. PCORI Methodology Committee. The PCORI methodology report. Pa­tient-Centered Outcomes Research Institute website.­sets/2013/11/PCORI-Methodology-Report.pdf. Published November 2013. Ac­cessed November 2015.

12. Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167.

13. Neuburger J, Hutchings A, van der Meulen J, Black N. Using patient-reported outcomes (PROs) to compare the providers of surgery: does the choice of measure matter? Med Care. 2013;51(6):517-523.

14. Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring pa­tient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17(2):179-193.
Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up