Currently Viewing:
The American Journal of Managed Care August 2018
Impact of a Medical Home Model on Costs and Utilization Among Comorbid HIV-Positive Medicaid Patients
Paul Crits-Christoph, PhD; Robert Gallop, PhD; Elizabeth Noll, PhD; Aileen Rothbard, ScD; Caroline K. Diehl, BS; Mary Beth Connolly Gibbons, PhD; Robert Gross, MD, MSCE; and Karin V. Rhodes, MD, MS
Choosing Wisely Clinical Decision Support Adherence and Associated Inpatient Outcomes
Andrew M. Heekin, PhD; John Kontor, MD; Harry C. Sax, MD; Michelle S. Keller, MPH; Anne Wellington, BA; and Scott Weingarten, MD
Precision Medicine and Sharing Medical Data in Real Time: Opportunities and Barriers
Y. Tony Yang, ScD, and Brian Chen, PhD, JD
Levers to Reduce Use of Unnecessary Services: Creating Needed Headroom to Enhance Spending on Evidence-Based Care
Michael Budros, MPH, MPP, and A. Mark Fendrick, MD
From the Editorial Board: Michael E. Chernew, PhD
Michael E. Chernew, PhD
Currently Reading
Optimizing Number and Timing of Appointment Reminders: A Randomized Trial
John F. Steiner, MD, MPH; Michael R. Shainline, MS, MBA; Jennifer Z. Dahlgren, MS; Alan Kroll, MSPT, MBA; and Stan Xu, PhD
Baseline and Postfusion Opioid Burden for Patients With Low Back Pain
Kevin L. Ong, PhD; Kirsten E. Stoner, PhD; B. Min Yun, PhD; Edmund Lau, MS; and Avram A. Edidin, PhD
Patient and Physician Predictors of Hyperlipidemia Screening and Statin Prescription
Sneha Kannan, MD; David A. Asch, MD, MBA; Gregory W. Kurtzman, BA; Steve Honeywell Jr, BS; Susan C. Day, MD, MPH; and Mitesh S. Patel, MD, MBA, MS
Evaluating HCV Screening, Linkage to Care, and Treatment Across Insurers
Karen Mulligan, PhD; Jeffrey Sullivan, MS; Lara Yoon, MPH; Jacki Chou, MPP, MPL; and Karen Van Nuys, PhD
Reducing Coprescriptions of Benzodiazepines and Opioids in a Veteran Population
Ramona Shayegani, PharmD; Mary Jo Pugh, PhD; William Kazanis, MS; and G. Lucy Wilkening, PharmD
Medicare Advantage Enrollees’ Use of Nursing Homes: Trends and Nursing Home Characteristics
Hye-Young Jung, PhD; Qijuan Li, PhD; Momotazur Rahman, PhD; and Vincent Mor, PhD

Optimizing Number and Timing of Appointment Reminders: A Randomized Trial

John F. Steiner, MD, MPH; Michael R. Shainline, MS, MBA; Jennifer Z. Dahlgren, MS; Alan Kroll, MSPT, MBA; and Stan Xu, PhD
Two text message or phone reminders were more effective in reducing missed primary care appointments than a single reminder, particularly in patients at high risk of missing appointments.
The patient-level predictors of missed appointments in Table 3 are similar to those found in prior studies.8,12-16 The C statistic of 0.93 indicates that the model was able to accurately discriminate between individuals who completed their appointments and those who missed them.19 Other organizations have also reported high predictive accuracy for such models.13,16 Although several sociodemographic and clinical variables contributed to the model, a model including just the number of prior missed appointments had a C statistic of 0.92, which suggests that attendance at prior appointments alone can accurately identify individuals at risk of subsequent missed visits.

Our analysis of high-risk patients was particularly informative. We developed a multivariable risk score rather than assessing intervention impact on multiple subgroups as defined by individual risk factors.20,22,23 This analysis showed that among 75% of patients, the risk of a missed appointment was less than 1% (Table 4). Although 2 reminders were slightly more effective than a single reminder in this group, the NNT of 1328 demonstrated that reminders for primary care visits were rarely beneficial. In contrast, the model identified a high-risk quartile of patients whose rate of missed appointments was 23%, comparable with rates reported in other settings.1,5 In this subgroup, 2 reminders were substantially more effective than 1 (NNT of 25), but they only reduced the rate of missed appointments to 20.5%. Automated text or phone messages will need to be augmented by more resource-intensive interventions, such as staff outreach or patient navigation,16,24 to further reduce missed appointments in these individuals.

Strengths and Limitations

The study had several strengths. It demonstrated that randomization can be used to rapidly address questions of operational importance using data from administrative and clinical sources. The marginal costs of the intervention were low because operational data, the IVR-T system, and the survey protocol were already in place. The results are consistent with those of other recent studies and systematic reviews2,5,14 and should be replicable in other healthcare systems that provide automated reminders by telephone or text message.

The study also had limitations, many of which illustrate the constraints of conducting randomized trials in an operational environment. Our intervention took place within a single integrated delivery system with a low baseline rate of missed appointments. Thus, our findings may not be generalizable to other settings. Fewer individuals were randomized into the trial than projected. Although the sample size could easily have been increased by extending the trial, we honored our original commitment to organizational leaders to complete the intervention within 2 months. Consequently, the number of completed surveys was also smaller than predicted, limiting statistical power to identify differences in satisfaction. Because only individuals who kept their appointments were included in the visit satisfaction survey, we could not assess whether those who missed or cancelled their visits had different attitudes about reminders. Additionally, although the response rate was not calculated in our study, about 35% of members typically respond to visit satisfaction surveys, raising concerns about response bias. The study was conducted over 2 months, and we did not assess the sustained effectiveness of the intervention. Some variables in our prediction model, such as the number of prior outpatient appointments, ED visits, and hospitalizations, may be more difficult to assess outside of integrated healthcare delivery systems. Although patients who made appointments online received email reminders in addition to IVR-T reminders, these individuals were likely distributed equally among the 3 study arms and did not affect the relative differences we observed. Finally, we did not determine whether appointments cancelled after the 3-day reminder were more often filled by other patients than those cancelled after the 1-day reminder.5,15


If other studies confirm that multiple reminders are more effective than a single reminder in reducing missed appointments, system leaders will face the decision about whether to send multiple reminders to all patients with upcoming appointments or restrict this approach to patients at highest risk. An argument in favor of the first approach is that multiple reminders did not appear to reduce visit satisfaction. However, if local satisfaction surveys suggest that patients view repeated reminders as unnecessary or intrusive or if an external IVR-T vendor bases their charges on the number of reminders sent, restriction of multiple reminders to high-risk patients might be justified despite the additional analytic effort necessary to implement the risk algorithm in real time.

In conclusion, we found that multiple text message or telephone reminders were more effective than a single reminder in reducing missed primary care visits, particularly in patients at highest risk. This study also demonstrates the value of a learning health system collaboration between operational leaders and researchers to address pragmatic questions of immediate concern.7

Author Affiliations: Institute for Health Research, Kaiser Permanente Colorado (JFS, MRS, SX), Denver, CO; Kaiser Foundation Health Plan of Colorado (JZD, AK), Denver, CO.

Source of Funding: This project was funded by the Kaiser Permanente Health Plan of Colorado.

Author Disclosures: Dr Steiner received internal grant support from Kaiser Permanente to conduct this study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (JFS, MRS, JZD, AK, SX); acquisition of data (JFS, MRS, JZD, SX); analysis and interpretation of data (JFS, MRS, SX); drafting of the manuscript (JFS, AK, SX); critical revision of the manuscript for important intellectual content (JFS, MRS, JZD); statistical analysis (SX); provision of patients or study materials (JFS, MRS); obtaining funding (JFS, AK); administrative, technical, or logistic support (AK); and supervision (JFS, SX).

Address Correspondence to: John F. Steiner, MD, MPH, Institute for Health Research, 10065 E Harvard Ave, Ste 300, Denver, CO 80231. Email:

1. Macharia WM, Leon G, Rowe BH, Stephenson BJ, Haynes RB. An overview of interventions to improve compliance with appointment keeping for medical services. JAMA. 1992;267(13):1813-1817. doi: 10.1001/jama.1992.03480130129038.

2. Free C, Phillips G, Watson L, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med. 2013;10(1):e1001363. doi: 10.1371/journal.pmed.1001363.

3. Peterson K, McCleery E, Anderson J, Waldrip K, Helfand M. Evidence brief: comparative effectiveness of appointment recall reminder procedures for follow-up appointments. In: VA Evidence-based Synthesis Program Evidence Briefs. Washington, DC: Department of Veterans Affairs (US); 2011.

4. Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J Telemed Telecare. 2011;17(7):358-364. doi: 10.1258/jtt.2011.110707.

5. McLean SM, Booth A, Gee M, et al. Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence. 2016;10:479-499. doi: 10.2147/PPA.S93046.

6. Olsen L, Aisner D, McGinnis JM, eds. The Learning Healthcare System: Workshop Summary. Washington, DC: The National Academies Press; 2007.

7. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207-210. doi: 10.7326/0003-4819-157-3-201208070-00012.

8. Steiner JF, Shainline MR, Bishop MC, Xu S. Reducing missed primary care appointments in a learning health system: two randomized trials and validation of a predictive model. Med Care. 2016;54(7):689-696. doi: 10.1097/MLR.0000000000000543.

9. Shoup JA, Madrid C, Koehler C, et al. Effectiveness and cost of influenza vaccine reminders for adults with asthma or chronic obstructive pulmonary disease. Am J Manag Care. 2015;21(7):e405-e413.

10. Kempe KL, Shetterly SM, France EK, Levin TR. Automated phone and mail population outreach to promote colorectal cancer screening. Am J Manag Care. 2012;18(7):370-378.

11. Bender BG, Cvietusa PJ, Goodrich GK, et al. Pragmatic trial of health care technologies to improve adherence to pediatric asthma treatment: a randomized clinical trial. JAMA Pediatr. 2015;169(4):317-323. doi: 10.1001/jamapediatrics.2014.3280.

12. Goldman L, Freidin R, Cook EF, Eigner J, Grich P. A multivariate approach to the prediction of no-show behavior in a primary care center. Arch Intern Med. 1982;142(3):563-567. doi: 10.1001/archinte.1982.00340160143026.

13. Lee VJ, Earnest A, Chen MI, Krishnan B. Predictors of failed attendances in a multi-specialty outpatient centre using electronic databases. BMC Health Serv Res. 2005;5:51. doi: 10.1186/1472-6963-5-51.

14. Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. Am J Med. 2010;123(6):542-548. doi: 10.1016/j.amjmed.2009.11.022.

15. Perron NJ, Dao MD, Kossovsky MP, et al. Reduction of missed appointments at an urban primary care clinic: a randomised controlled study. BMC Fam Pract. 2010;11:79. doi: 10.1186/1471-2296-11-79.

16. Percac-Lima S, Cronin PR, Ryan DP, Chabner BA, Daly EA, Kimball AB. Patient navigation based on predictive modeling decreases no-show rates in cancer care. Cancer. 2015;121(10):1662-1670. doi: 10.1002/cncr.29236.

17. Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130-1139. doi: 10.1097/01.mlr.0000182534.19832.83.

18. Jha AK, Orav EJ, Zheng J, Epstein AM. Patients’ perception of hospital care in the United States. N Engl J Med. 2008;359(18):1921-1931. doi: 10.1056/NEJMsa0804116.

19. Meurer WJ, Tolles J. Logistic regression diagnostics: understanding how well a model predicts outcomes. JAMA. 2017;317(10):1068-1069. doi: 10.1001/jama.2016.20441.

20. Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JP. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Intern Med. 2017;177(4):554-560. doi: 10.1001/jamainternmed.2016.9125.

21. Griffin JM, Hulbert EM, Vernon SW, et al. Improving endoscopy completion: effectiveness of an interactive voice response system. Am J Manag Care. 2011;17(3):199-208.

22. Hayward RA, Kent DM, Vijan S, Hofer TP. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006;6:18. doi: 10.1186/1471-2288-6-18.

23. Burke JF, Hayward RA, Nelson JP, Kent DM. Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials. Circ Cardiovasc Qual Outcomes. 2014;7(1):163-169. doi: 10.1161/CIRCOUTCOMES.113.000497.

24. Shah SJ, Cronin P, Hong CS, et al. Targeted reminder phone calls to patients at high risk of no-show for primary care appointment: a randomized trial. J Gen Intern Med. 2016;31(12):1460-1466. doi: 10.1007/s11606-016-3813-0.
Copyright AJMC 2006-2020 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up