Currently Viewing:
The American Journal of Managed Care August 2018
Impact of a Medical Home Model on Costs and Utilization Among Comorbid HIV-Positive Medicaid Patients
Paul Crits-Christoph, PhD; Robert Gallop, PhD; Elizabeth Noll, PhD; Aileen Rothbard, ScD; Caroline K. Diehl, BS; Mary Beth Connolly Gibbons, PhD; Robert Gross, MD, MSCE; and Karin V. Rhodes, MD, MS
Currently Reading
Choosing Wisely Clinical Decision Support Adherence and Associated Inpatient Outcomes
Andrew M. Heekin, PhD; John Kontor, MD; Harry C. Sax, MD; Michelle S. Keller, MPH; Anne Wellington, BA; and Scott Weingarten, MD
Levers to Reduce Use of Unnecessary Services: Creating Needed Headroom to Enhance Spending on Evidence-Based Care
Michael Budros, MPH, MPP, and A. Mark Fendrick, MD
From the Editorial Board: Michael E. Chernew, PhD
Michael E. Chernew, PhD
Optimizing Number and Timing of Appointment Reminders: A Randomized Trial
John F. Steiner, MD, MPH; Michael R. Shainline, MS, MBA; Jennifer Z. Dahlgren, MS; Alan Kroll, MSPT, MBA; and Stan Xu, PhD
Impact of After-Hours Telemedicine on Hospitalizations in a Skilled Nursing Facility
David Chess, MD; John J. Whitman, MBA; Diane Croll, DNP; and Richard Stefanacci, DO
Baseline and Postfusion Opioid Burden for Patients With Low Back Pain
Kevin L. Ong, PhD; Kirsten E. Stoner, PhD; B. Min Yun, PhD; Edmund Lau, MS; and Avram A. Edidin, PhD
Patient and Physician Predictors of Hyperlipidemia Screening and Statin Prescription
Sneha Kannan, MD; David A. Asch, MD, MBA; Gregory W. Kurtzman, BA; Steve Honeywell Jr, BS; Susan C. Day, MD, MPH; and Mitesh S. Patel, MD, MBA, MS
Evaluating HCV Screening, Linkage to Care, and Treatment Across Insurers
Karen Mulligan, PhD; Jeffrey Sullivan, MS; Lara Yoon, MPH; Jacki Chou, MPP, MPL; and Karen Van Nuys, PhD
Reducing Coprescriptions of Benzodiazepines and Opioids in a Veteran Population
Ramona Shayegani, PharmD; Mary Jo Pugh, PhD; William Kazanis, MS; and G. Lucy Wilkening, PharmD
Medicare Advantage Enrollees’ Use of Nursing Homes: Trends and Nursing Home Characteristics
Hye-Young Jung, PhD; Qijuan Li, PhD; Momotazur Rahman, PhD; and Vincent Mor, PhD

Choosing Wisely Clinical Decision Support Adherence and Associated Inpatient Outcomes

Andrew M. Heekin, PhD; John Kontor, MD; Harry C. Sax, MD; Michelle S. Keller, MPH; Anne Wellington, BA; and Scott Weingarten, MD
This analysis examines the associations between adherence to Choosing Wisely recommendations embedded into clinical decision support alerts and 4 measures of resource use and quality.

One limitation is our strict definition of “alert compliance”: In order to be in the adherent encounter group, providers had to be adherent to all of the CW CDS-related alerts. Patient episodes in which clinicians followed some but not all of the CW alerts that fired were considered “mixed-adherence” episodes and were excluded from analysis. This strict inclusion criterion limits our understanding of the clinical and financial impacts that patients with partially adherent episodes may have experienced. Similarly, we were unable to differentiate the impact of specific alerts on our studied outcomes. Although there appear to be some differences between individual alerts, our study did not have enough power to make inferences due to its small sample sizes. Another key limitation of this study is the lack of control for provider effects. The analysis did not include provider characteristics and thus could not examine confounding on the provider level; it is possible that some providers are more likely to trigger alerts or are more likely to be nonadherent to alerts, even though we found no overall correlation between provider acceptance rate and provider outcomes. Additionally, providers who are more likely to adhere to evidence-based guidelines, including CW, may be more likely to ascribe to other system-based approaches and practices consistent with value-based patient care. We need to better understand the differences in characteristics and practice patterns of providers who adhere to CW recommendations compared with providers who do not.

Future analyses should examine the role of specific physician and alert characteristics on adherence to CDS and the effects on outcomes. The Elixhauser index computation was based on all relevant diagnoses made on all encounters for a given patient within our data set. It is possible, however, that the patient could have received additional relevant diagnoses outside of the time frame or hospital system applicable to this study.

Although our regression models adjusted for severity of illness, it is possible that the model did not control for all differences in patient severity or characteristics. Moreover, this study did not seek to establish causation between CW adherence and improved patient and financial outcomes. Many factors determine whether a single alert is adhered to or ignored, including alert fatigue,38 provider familiarity with the guideline presented,39 fear of malpractice,13 or need to reassure one’s patient through further diagnostic tests.16 We were unable to capture some relevant data, including professional billing fees and cost and readmissions data from other facilities, limiting our outcomes analyses. Finally, our demonstrated correlations between adherence and outcomes cannot necessarily be generalized to all CDS interventions, as the alerts evaluated in this study were implemented in the inpatient setting, were deemed the most technically feasible to deploy accurately, and had sufficient volume to evaluate.


We recommend that health systems consider real-time CDS interventions as a method to encourage improved adoption of CW and other evidence-based guidelines. A meta-analysis of CDS systems concluded that by providing context-specific information at the point of care, the odds of providers adopting guideline recommendations are 112 times higher.19 CDS enables the provision of context-specific information at the point of care and could help to overcome several known barriers to CW guideline adoption.

Our findings contribute to the evidence base surrounding the use of CDS and improvements in patient clinical and financial outcomes. Formal prospective cohort studies and randomized CDS intervention trials, perhaps randomizing providers assigned to receive CDS interventions, should be prioritized to help guide future provider strategies in regard to reducing low-value care. 


The authors gratefully acknowledge the administrative and material support of Georgia Hoyler and Marin Lopci.

Author Affiliations: Optum (AMH, JK), Washington, DC; Cedars-Sinai Medical Center (HCS, MSK, SW), Los Angeles, CA; Stanson Health (AW, SW), Los Angeles, CA.

Source of Funding: No funding beyond existing employment agreements was provided for this research.

Author Disclosures: Drs Heekin and Kontor are employed by Optum, which is a licensed reseller of Stanson Health, including its Choosing Wisely alert content evaluated in this study. Dr Sax is employed by Cedars-Sinai, which is the major shareholder of Stanson Health; he is not directly involved in or on the board of Stanson. Ms Keller is employed by Cedars-Sinai Medical Center, which is the employer of Stanson Health’s founders Scott Weingarten and Darren Dworkin. Ms Wellington was employed by and owns stock in Stanson Health; she is now employed at Cedars-Sinai Medical Center, which is a major shareholder of Stanson Health. Dr Weingarten is Chairman of the Board of and owns stock in Stanson Health.

Authorship Information: Concept and design (AMH, JK, HCS, SW); acquisition of data (AW); analysis and interpretation of data (AMH, JK, HCS, MSK, AW, SW); drafting of the manuscript (AMH, JK, HCS, MSK, AW, SW); critical revision of the manuscript for important intellectual content (JK, HCS, MSK, SW); statistical analysis (AMH); and provision of patients or study materials (SW).

Address Correspondence to: John Kontor, MD, Optum, 2445 M St NW, Washington, DC 20001. Email:

1. Kayyali B, Knott D, Van Kuiken S. The big-data revolution in US health care: accelerating value and innovation. McKinsey & Company website. Published April 2013. Accessed December 2, 2016.

2. Hillestad R, Bigelow J, Bower A, et al. Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health Aff (Millwood). 2005;24(5):1103-1117. doi: 10.1377/hlthaff.24.5.1103.

3. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA. 1993;269(3):379-383. doi: 10.1001/jama.1993.03500030077036.

4. Wang SJ, Middleton B, Prosser LA, et al. A cost-benefit analysis of electronic medical records in primary care. Am J Med. 2003;114(5):397-403. doi: 10.1016/S0002-9343(03)00057-3.

5. Henry J, Pylypchuk Y, Searcy T, Patel V. Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008-2015 [ONC data brief no. 35]. Office of the National Coordinator for Health Information Technology website. Published May 2016. Accessed October 27, 2016.

6. Adler-Milstein J, Everson J, Lee S-YD. EHR adoption and hospital performance: time-related effects. Health Serv Res. 2015;50(6):1751-1771. doi: 10.1111/1475-6773.12406.

7. Kern LM, Barrón Y, Dhopeshwarkar RV, Edwards A, Kaushal R; HITEC Investigators. Electronic health records and ambulatory quality of care. J Gen Intern Med. 2013;28(4):496-503. doi: 10.1007/s11606-012-2237-8.

8. Campanella P, Lovato E, Marone C, et al. The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. Eur J Public Health. 2016;26(1):60-64. doi: 10.1093/eurpub/ckv122.

9. Lammers EJ, McLaughlin CG, Barna M. Physician EHR adoption and potentially preventable hospital admissions among Medicare beneficiaries: panel data evidence, 2010-2013. Health Serv Res. 2016;51(6):2056-2075. doi: 10.1111/1475-6773.12586.

10. Patterson ME, Marken P, Zhong Y, Simon SD, Ketcherside W. Comprehensive electronic medical record implementation levels not associated with 30-day all-cause readmissions within Medicare beneficiaries with heart failure. Appl Clin Inform. 2014;5(3):670-684. doi: 10.4338/aci-2014-01-ra-0008.

11. Wolfson D, Santa J, Slass L. Engaging physicians and consumers in conversations about treatment overuse and waste: a short history of the Choosing Wisely campaign. Acad Med. 2014;89(7):990-995. doi: 10.1097/ACM.0000000000000270.

12. Berwick DM, Hackbarth AD. Eliminating waste in US health care. JAMA. 2012;307(14):1513-1516. doi: 10.1001/jama.2012.362.

13. Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the Choosing Wisely campaign. JAMA Intern Med. 2015;175(12):1913-1920. doi: 10.1001/jamainternmed.2015.5441.

14. Reid RO, Rabideau B, Sood N. Low-value health care services in a commercially insured population. JAMA Intern Med. 2016;176(10):1567-1571. doi: 10.1001/jamainternmed.2016.5031.

15. Zikmund-Fisher BJ, Kullgren JT, Fagerlin A, Klamerus ML, Bernstein SJ, Kerr EA. Perceived barriers to implementing individual Choosing Wisely recommendations in two national surveys of primary care providers. J Gen Intern Med. 2017;32(2):210-217. doi: 10.1007/s11606-016-3853-5.

16. Krouss M, Croft L, Morgan DJ. Physician understanding and ability to communicate harms and benefits of common medical treatments. JAMA Intern Med. 2016;176(10):1565-1567. doi: 10.1001/jamainternmed.2016.5027.

17. Colla CH, Kinsella EA, Morden NE, Meyers DJ, Rosenthal MB, Sequist TD. Physician perceptions of Choosing Wisely and drivers of overuse. Am J Manag Care. 2016;22(5):337-343.

18. Clinical decision support: more than just ‘alerts’ tipsheet. CMS website. Published 2014. Accessed October 25, 2016.

19. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi: 10.1136/bmj.38398.500764.8F.

20. Goldzweig CL, Orshansky G, Paige NM, et al. Electronic health record-based interventions for improving appropriate diagnostic imaging: a systematic review and meta-analysis. Ann Intern Med. 2015;162(8):557-565. doi: 10.7326/m14-2600.

21. Raja AS, Gupta A, Ip IK, Mills AM, Khorasani R. The use of decision support to measure adherence to a national imaging quality measure. Acad Radiol. 2014;21(3):378-383. doi: 10.1016/j.acra.2013.10.017.

22. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8-27.

23. Clinical Classifications Software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project website. Published October 2016. Accessed December 2, 2016.
24. Lawthers AG, McCarthy EP, Davis RB, Peterson LE, Palmer RH, Iezzoni LI. Identification of in-hospital complications from claims data: is it valid? Med Care. 2000;38(8):785-795.

25. Readmissions Reduction Program (HRRP). CMS website. Published April 2016. Accessed December 2, 2016.

26. Hospital-acquired conditions. CMS website. Published August 2015. Accessed December 2, 2016.

27. The R project for statistical computing. R project website. Published 2016. Accessed August 16, 2016.

28. glm2: fitting generalized linear models. R project website. Published 2014. Accessed August 16, 2016.

29. caret: classification and regression training. R project website. Published 2016. Accessed August 16, 2016.

30. sqldf: manipulate R data frames using SQL. R project website. Published 2014. Accessed August 16, 2016.

31. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc. 2009;16(4):531-538. doi: 10.1197/jamia.M2910.

32. Dimagno MJ, Wamsteker EJ, Rizk RS, et al. A combined paging alert and web-based instrument alters clinician behavior and shortens hospital length of stay in acute pancreatitis. Am J Gastroenterol. 2014;109(3):306-315. doi: 10.1038/ajg.2013.282.

33. Vicente V, Svensson L, Wireklint Sundström B, Sjöstrand F, Castren M. Randomized controlled trial of a prehospital decision system by emergency medical services to ensure optimal treatment for older adults in Sweden. J Am Geriatr Soc. 2014;62(7):1281-1287. doi: 10.1111/jgs.12888.

34. Panella M, Marchisio S, Di Stanislao F. Reducing clinical variations with clinical pathways: do pathways work? Int J Qual Health Care. 2003;15(6):509-521. doi: 10.1093/intqhc/mzg057.

35. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23(4):451-458. doi: 10.1007/s11606-008-0504-5.

36. Lin Y-C, Chang C-S, Yeh C-J, Wu Y-C. The appropriateness and physician compliance of platelet usage by a computerized transfusion decision support system in a medical center. Transfusion. 2010;50(12):2565-2570. doi: 10.1111/j.1537-2995.2010.02757.x.

37. Bayati M, Braverman M, Gillam M, et al. Data-driven decisions for reducing readmissions for heart failure: general methodology and case study. PLoS One. 2014;9(10):e109264. doi: 10.1371/journal.pone.0109264.

38. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138-147. doi: 10.1197/jamia.M1809.

39. Halm EA, Atlas SJ, Borowsky LH, et al. Understanding physician adherence with a pneumonia practice guideline: effects of patient, system, and physician factors. Arch Intern Med. 2000;160(1):98-104. doi: 10.1001/archinte.160.1.98.
Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up