Currently Viewing:
The American Journal of Managed Care May 2019
Evaluation of Value-Based Insurance Design for Primary Care
Qinli Ma, PhD; Gosia Sylwestrzak, MA; Manish Oza, MD; Lorraine Garneau; and Andrea R. DeVries, PhD
The Presurgical Episode: An Untapped Opportunity to Improve Value
Erika D. Sears, MD, MS; Rodney A. Hayward, MD; and Eve A. Kerr, MD, MPH
Clarification of References to Medication Adherence Scale
Open Doors to Primary Care Should Add a “Screen” to Reduce Low-Value Care
Betsy Q. Cliff, MS; and A. Mark Fendrick, MD
From the Editorial Board: Daniel B. Wolfson, MHSA
Daniel B. Wolfson, MHSA
Cost-Effectiveness of DPP-4 Inhibitor and SGLT2 Inhibitor Combination Therapy for Type 2 Diabetes
Manjiri Pawaskar, PhD; S. Pinar Bilir, MS; Stacey Kowal, MS; Claudio Gonzalez, MD; Swapnil Rajpathak, MD; and Glenn Davies, DrPH
Improving Provider Directory Accuracy: Can Machine-Readable Directories Help?
Michael Adelberg, MA, MPP; Austin Frakt, PhD; Daniel Polsky, PhD; and Michelle Kitchman Strollo, DrPH, MHS
Electronic Consults for Improving Specialty Care Access for Veterans
David E. Winchester, MD, MS; Anita Wokhlu, MD; Juan Vilaro, MD; Anthony A. Bavry, MD, MPH; Ki Park, MD; Calvin Choi, MD; Mark Panna, MD; Michael Kaufmann, MD; Matthew McKillop, MD; and Carsten Schmalfuss, MD
Potential Impact of Pharmaceutical Industry Rebates on Medication Adherence
Leah L. Zullig, PhD; Bradi B. Granger, PhD; Helene Vilme, DrPH; Megan M. Oakes, MPA; and Hayden B. Bosworth, PhD
Producing Comparable Cost and Quality Results From All-Payer Claims Databases
Maria de Jesus Diaz-Perez, PhD; Rita Hanover, PhD; Emilie Sites, MPH; Doug Rupp, BS; Jim Courtemanche, MS; and Emily Levi, MPH
Currently Reading
Beyond Satisfaction Scores: Exploring Emotionally Adverse Patient Experiences
Laura M. Holdsworth, PhD; Dani L. Zionts, MScPH; Karen Marie De Sola-Smith, PhD; Melissa Valentine, PhD; Marcy D. Winget, PhD; and Steven M. Asch, MD
Pilot of Urgent Care Center Evaluation for Acute Coronary Syndrome
Ryan P. Radecki, MD, MS; Kevin F. Foley, PhD; Timothy S. Elzinga, MD; Cynthia P. Horak, MD; Thomas E. Gant, MS; Heather M. Papp, BA; Adam J. Morris, BS; Natalie R. Hauser, BA; and Briar L. Ertz-Berger, MD, MPH

Beyond Satisfaction Scores: Exploring Emotionally Adverse Patient Experiences

Laura M. Holdsworth, PhD; Dani L. Zionts, MScPH; Karen Marie De Sola-Smith, PhD; Melissa Valentine, PhD; Marcy D. Winget, PhD; and Steven M. Asch, MD
This study explores the causes of emotionally adverse patient experiences in cancer care and presents a taxonomy for analyzing free-text patient data.

This study has identified that although emotionally adverse experiences are infrequent, variability exists in how patients express these experiences, as well as their causes and predictors and their consequences. Studying oncology patients was a strength, as this population generally experiences more complex, long-term care requiring greater coordination with high emotional valence, especially compared with episodic care. Although the vast majority of patient survey comments were positive or neutral, patients expressed strong negative emotion in a range of ways, such as through sarcasm, hyperbole, and rhetorical statements. Methodologically, the intensity was variable across issues between interview and survey responses. For example, when patients talked about scheduling problems in interviews, they often spoke with great intensity, and these problems were rated as emotionally adverse experiences. Additionally, the interactive nature of the interview meant that interviewers could probe for information about negative experiences. In survey comments, some wording implied negativity (eg, “later,” “not”), but actual feelings of negativity were not expressed. It may be that these patients were concerned about potential negative impacts to their care as a result of voicing displeasure, as they completed the surveys in the clinic setting and therefore may have perceived that their responses might be identifiable.29

Some patients expressed their feelings in nuanced ways that might be difficult for health systems to identify using typical methods of patient satisfaction surveys or complaint records. Few patients talked about making a formal complaint. Health systems could use existing data sources, such as the Hospital Consumer Assessment of Healthcare Providers and Systems survey comments, to identify negative experiences, but analyzing these data sensitively is time  intensive and cumbersome. It should be noted that the cancer center in which this study took place had a high Likelihood to Recommend score at the same time as the study (87.2%, in the 91st percentile nationally; C. Montalvo, BA, written communication, April 2018) and high scores for the survey on which the patient comments were written,26 indicating that overall there is a high level of satisfaction among patients treated at this institution. However, as this study indicates, focusing on survey scores alone may miss critiques that afford opportunities for improvement even in a highly rated system. We believe that health systems would benefit from analyzing textual data to ensure that responses to quality issues are congruent with patients’ priorities in care.2 Our system of identifiers could be used as a categorization system for such data.

Although identifying and defining emotionally adverse experiences was more challenging than expected, the range of triggers was less surprising. There was wide variation in the triggers, but threats to well-being and trust were almost universally an affront to patients. This aligns with existing research that has found that causes of acute disgust have in common dehumanizing experiences and breaches of trust13,30 but that thresholds for tolerating such feelings and coping are variable within and across individuals. In contrast to the literature, some themes we found, such as wait times and travel issues, do not appear to be related to well-being or trust but might reflect other underlying issues that precondition patients’ sensitivity to such annoyances over time.25 For example, if confidence in the competence of the care provider is undermined, patients might be inclined to look for or detect other lapses in care. We found the inverse to be true: An excellent care provider could reduce the burden of annoyances. In this way, emotionally adverse experiences are formed within a context of priorities; the overall priority of surviving cancer might make patients tolerate more than they would otherwise. Alternatively, patients with a serious illness may feel that they are already under stress and perceive typical low-level annoyances as a serious threat. The relative and temporal nature of emotionally adverse experiences that evolve with time in a healthcare context is not a feature of consumer disgust as described in the marketing literature,12 although the accumulation of experience has been acknowledged as a feature of patient satisfaction.31 The conceptual map presented in the Figure may help further understanding of the components of patient satisfaction and particularly the “process” aspect of patient satisfaction with care.31

Although some approaches to quality improvement might focus on enhancing positive attributes of care, such as through appreciative inquiry,32 our framework identified important domains that have typically been absent from predictors of patient satisfaction,8 including travel or transportation, education and information, scheduling, and finance/insurance. Likewise, typical indicators found in patient satisfaction models that emphasize positive attributes, such as the environment and physical setting, were not present in our framework.8,33 There may not be complete congruence between issues that positively and negatively influence satisfaction. More mundane features of care, like travel to appointments and scheduling, might be noticed only when they fail to go smoothly, and they therefore might be overlooked by quality improvement that focuses on positive aspects only. Dissatisfaction with care may be more telling than satisfaction,31 particularly if patient outcomes are adversely affected. Our aim is not to enumerate absolute triggers of disgust but, rather, to describe the range of patient-specific issues that can trigger such feelings and find ways to recognize them, as we perceive that this may help health systems identify opportunities for quality improvement. Indeed, the medical center in this study responded positively to identifying adverse patient experiences and used it as an opportunity to target improvements in care.


We developed our concept of emotionally adverse experiences using Fortini-Campbell’s framework12 as those that are both important and negative, but within negative could be a range of emotions that were difficult to differentiate. Emotions such as disgust, anger, and fear are universal in the cancer experience and not always related to dissatisfaction with care. To protect patient privacy, we were unable to link surveys to data in the electronic health record for collecting demographic information. Although we have limited demographic information on interview participants, we perceived that the age and gender mix of our sample was broadly reflective of the patient population of the cancer center. For both surveys and interviews, non-English speakers were likely underrepresented. We did not have access to all the interview audio files, as this was a secondary analysis of data. Spoken data may provide more clues to patients’ emotional states, although listening to the audio did not change our interpretation of transcripts. The survey distributed to patients covered 5 theme areas, which may have constrained patients’ comments to these content areas, therefore potentially missing other adverse experiences. However, the themes covered a range of experiences and patients were not instructed to limit their comments. Indeed, many chose to write about topics not specifically queried in the survey. Our findings are limited to the experience of 1 institution.


We present a categorization system for adverse patient experiences that can be applied to qualitative data, like free-text survey comments, even when satisfaction ratings are high. The 10 domains demonstrate a wide range of issues that can lead to emotionally adverse experiences, which could be difficult for health systems to tackle at once. Drawing on the specificity found in routinely collected qualitative data, such as survey comments, can help target quality improvement efforts to those domains in greatest need of improvement. Further research should be conducted to test the congruence of extreme dissatisfiers with extreme delighters in healthcare. In the meantime, listening to the dissatisfied patient voice in survey comments can help providers and managers alike improve care, even in high-performing systems.


The authors thank Gurpreet Ishpuniani, BS, of Stanford University, and the patients, family caregivers, staff, and administrators for their contributions to this study.

Author Affiliations: Division of Primary Care and Population Health, Stanford University School of Medicine (LMH, DLZ, MDW, SMA), Stanford, CA; Betty Irene Moore School of Nursing, University of California, Davis (KMDS-S), Sacramento, CA; Department of Management Science and Engineering, School of Engineering, Stanford University (MV), Stanford, CA.

Source of Funding: Stanford Health Care.

Author Disclosures: Dr De Sola-Smith received a modest payment for completion of the background and discussion sections of the manuscript. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (LMH, MDW, SMA); acquisition of data (LMH, KMDS-S, MV); analysis and interpretation of data (LMH, DLZ, MDW); drafting of the manuscript (LMH, DLZ, KMDS-S, MDW, SMA); critical revision of the manuscript for important intellectual content (LMH, DLZ, KMDS-S, MV, MDW, SMA); statistical analysis (DLZ, MDW); provision of patients or study materials (MV, MDW); obtaining funding (MDW, SMA); administrative, technical, or logistic support (DLZ); and supervision (MDW).

Address Correspondence to: Laura M. Holdsworth, PhD, Division of Primary Care and Population Health, Stanford University School of Medicine, 1265 Welch Rd, Stanford, CA 94305. Email:

1. Jones TO, Sasser WE Jr. Why satisfied customers defect. Harv Bus Rev. 1995;73(6):88-99. Accessed June 21, 2018.

2. Manary MP, Boulding W, Staelin R, Glickman SW. The patient experience and health outcomes. N Engl J Med. 2013;368(3):201-203. doi: 10.1056/NEJMp1211775.

3. Arora NK. Interacting with cancer patients: the significance of physicians’ communication behavior. Soc Sci Med. 2003;57(5):791-806. doi: 10.1016/S0277-9536(02)00449-5.

4. Street RL Jr, Makoul G, Arora NK, Epstein RM. How does communication heal? pathways linking clinician-patient communication to health outcomes. Patient Educ Couns. 2009;74(3):295-301. doi: 10.1016/j.pec.2008.11.015.

5. Street RL Jr, Mazor KM, Arora NK. Assessing patient-centered communication in cancer care: measures for surveillance of communication outcomes. J Oncol Pract. 2016;12(12):1198-1202. doi: 10.1200/JOP.2016.013334.

6. Attree M. Patients’ and relatives’ experiences and perspectives of “good” and “not so good” quality care. J Adv Nurs. 2001;33(4):456-466. doi: 10.1046/j.1365-2648.2001.01689.x.

7. Kvåle K, Bondevik M. What is important for patient centered care? a qualitative study about the perceptions of patients with cancer. Scand J Caring Sci. 2008;22(4):582-589. doi: 10.1111/j.1471-6712.2007.00579.x.

8. Shirley ED, Sanders JO. Patient satisfaction: implications and predictors of success. J Bone Joint Surg Am. 2013;95(10):e69. doi: 10.2106/JBJS.L.01048.

9. Heerdegen ACS, Petersen GS, Jervelund SS. Determinants of patient satisfaction with cancer care delivered by the Danish healthcare system. Cancer. 2017;123(15):2918-2926. doi: 10.1002/cncr.30673.

10. Malcolm E, Milstein A. Achieving higher quality and lower costs via innovations in healthcare delivery design. In: Phillips RA, ed. America’s Healthcare Transformation: Strategies and Innovations. New Brunswick, NJ: Rutgers University Press Medicine; 2016:105-112.

11. O’Toole K. How do you come up with good ideas? Stanford Bus. 2013;81(1):8-10. Accessed June 21, 2018.

12. Fortini-Campbell L. Integrated marketing and the consumer experience. In: Iacobucci D, Calder B, eds. Kellogg on Integrated Marketing. Hoboken, NJ: John Wiley & Sons Inc; 2003:54-89.

13. Reynolds LM, Bissett IP, Porter D, Consedine NS. The “ick” factor matters: disgust prospectively predicts avoidance in chemotherapy patients. Ann Behav Med. 2016;50(6):935-945. doi: 10.1007/s12160-016-9820-x.

14. Kannan VD, Veazie PJ. Predictors of avoiding medical care and reasons for avoidance behavior. Med Care. 2014;52(4):336-345. doi: 10.1097/MLR.0000000000000100.

15. Barry MJ, Edgeman-Levitan S. Shared decision making—the pinnacle of patient-centered care. N Engl J Med. 2012;366(9):780-781. doi: 10.1056/NEJMp1109283.

16. Institute of Medicine. Partnering With Patients to Drive Shared Decisions, Better Value, and Care Improvement. Washington, DC: The National Academies Press; 2014.

17. Bar-Sela G, Yochpaz S, Gruber R, Lulav-Grinwald D, Mitnik I, Koren D. The association between the strength of the working alliance and sharing concerns by advanced cancer patients: a pilot study. Support Care Cancer. 2016;24(1):319-325. doi: 10.1007/s00520-015-2794-6.

18. Dwamena F, Holmes-Rovner M, Gaulden CM, et al. Interventions for providers to promote a patient-centred approach in clinical consultations. Cochrane Database Syst Rev. 2012;12:CD003267. doi: 10.1002/14651858.CD003267.pub2.

19. Jerant A, Fenton JJ, Bertakis KD, Franks P. Satisfaction with health care providers and preventive care adherence. Med Care. 2014;52(1):78-85. doi: 10.1097/MLR.0000000000000021.

20. Zolnierek KB, Dimatteo MR. Physician communication and patient adherence to treatment: a meta-analysis. Med Care. 2009;47(8):826-834. doi: 10.1097/MLR.0b013e31819a5acc.

21. Bertakis KD, Azari R. Patient-centered care is associated with decreased health care utilization. J Am Board Fam Med. 2011;24(3):229-239. doi: 10.3122/jabfm.2011.03.100170.

22. Roberson Barnard S. Is it OK to fire my oncologist? J Oncol Pract. 2014;10(2):151-153. doi: 10.1200/JOP.2013.001243.

23. Gould J, Sinding C, Mitchell T, et al. “Below their notice”: exploring women’s subjective experiences of cancer system exclusion. J Cancer Educ. 2009;24(4):308-314. doi: 10.1080/08858190902997324.

24. Mathews M, Ryan D, Bulman D. What does satisfaction with wait times mean to cancer patients? BMC Cancer. 2015;15:1017. doi: 10.1186/s12885-015-2041-z.

25. Izugami S, Takase K. Consumer perception of inpatient medical services. PLoS One. 2016;11(11):e0166117. doi: 10.1371/journal.pone.0166117.

26. Winget M, Haji-Sheikhi F, Asch SM. Development of a tailored survey to evaluate a patient-centered initiative. Am J Manag Care. 2018;24(2):e294-e301.

27. Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. 2nd ed. London, United Kingdom: Sage; 1994.

28. Russell JA, Mehrabian A. Evidence for a three-factor theory of emotions. J Res Pers. 1977;11(3):273-294. doi: 10.1016/0092-6566(77)90037-X.

29. Hawkins JB, Brownstein JS, Tuli G, et al. Measuring patient-perceived quality of care in US hospitals using Twitter. BMJ Qual Saf. 2016;25(6):404-413. doi: 10.1136/bmjqs-2015-004309.

30. Hack TF, Degner LF, Parker PA; SCRN Communication Team. The communication goals and needs of cancer patients: a review. Psychooncology. 2005;14(10):831-845; discussion 846-847. doi: 10.1002/pon.949.

31. Sitzia J, Wood N. Patient satisfaction: a review of issues and concepts. Soc Sci Med. 1997;45(12):1829-1843. doi: 10.1016/S0277-9536(97)00128-7.

32. Moorer K, Kunupakaphun S, Delgado E, et al. Using appreciative inquiry as a framework to enhance the patient experience. Patient Exp J. 2017;4(3):128-135. Accessed June 21, 2018.

33. Naidu A. Factors affecting patient satisfaction and healthcare quality. Int J Health Care Qual Assur. 2009;22(4):366-381. doi: 10.1108/09526860910964834.
Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up