Currently Viewing:
The American Journal of Managed Care July 2015
Low-Value Care for Acute Sinusitis Encounters: Who's Choosing Wisely?
Adam L. Sharp, MD, MS; Marc H. Klau, MD, MBA; David Keschner, MD, JD; Eric Macy, MD, MS; Tania Tang, PhD, MPH; Ernest Shen, PhD; Corrine Munoz-Plaza, MPH; Michael Kanter, MD; Matthew A. Silver, MD;
Usefulness of Pharmacy Claims for Medication Reconciliation in Primary Care
Dominique Comer, PharmD, MS; Joseph Couto, PharmD, MBA; Ruth Aguiar, BA; Pan Wu, PhD; and Daniel J. Elliott, MD, MSCE
No Longer a Unicorn: Improving Health Through Accountable Care Organizations
Risa Lavizzo-Mourey, MD, MBA, president and CEO, The Robert Wood Johnson Foundation
ACA-Mandated Elimination of Cost Sharing for Preventive Screening Has Had Limited Early Impact
Shivan J. Mehta, MD, MBA; Daniel Polsky, PhD; Jingsan Zhu, MBA; James D. Lewis, MD, MSCE; Jonathan T. Kolstad, PhD; George Loewenstein, PhD; and Kevin G. Volpp, MD, PhD
Determinants of Medicare Plan Choices: Are Beneficiaries More Influenced by Premiums or Benefits?
Paul D. Jacobs, PhD; and Melinda B. Buntin, PhD
Acupuncture and Chiropractic Care: Utilization and Electronic Medical Record Capture
Charles Elder, MD, MPH; Lynn DeBar, PhD, MPH; Cheryl Ritenbaugh, PhD, MPH; William Vollmer, PhD; Richard A. Deyo, MD, MPH; John Dickerson, PhD; and Lindsay Kindler, PhD, RN
Worksite Medical Home: Health Services Use and Claim Costs
Christopher Conover, PhD; Rebecca Namenek Brouwer, MS; Gale Adcock, MSN, RN, FNP-BC, FAANP; David Olaleye, PhD, MSCe; John Shipway, BS; and Truls Østbye, MD, PhD
Currently Reading
Effectiveness and Cost of Influenza Vaccine Reminders for Adults With Asthma or Chronic Obstructive Pulmonary Disease
Jo Ann Shoup, MS; Carlos Madrid, MA; Caroline Koehler, RN, MSN; Cynthia Lamb, BS, RN; Jennifer Ellis, MSPH; Debra P. Ritzwoller, PhD; and Matthew F. Daley, MD
It's Time for CMS to Utilize Functional Assessment as a Key Quality Component
Wayne Matthews, MS, PA-C, DFAAPA
Colorectal Cancer Screening in the 21st Century: Where Do We Go From Here?
Jennifer K. Maratt, MD; and Sameer D. Saini, MD, MS
Pilot of Decision Support to Individualize Colorectal Cancer Screening Recommendations
Amy R. Schwartz, MD; Forrest L. Levin, MS; Joseph A. O’Neil, Jr, BS; and R. Scott Braithwaite, MD, MSc

Effectiveness and Cost of Influenza Vaccine Reminders for Adults With Asthma or Chronic Obstructive Pulmonary Disease

Jo Ann Shoup, MS; Carlos Madrid, MA; Caroline Koehler, RN, MSN; Cynthia Lamb, BS, RN; Jennifer Ellis, MSPH; Debra P. Ritzwoller, PhD; and Matthew F. Daley, MD
Interactive voice response reminders had neither a positive nor a negative effect on promoting influenza vaccination over reminders via postcards, but are a potentially less expensive option.
A total of 12,428 adults aged 19 to 64 years were identified with asthma or COPD. As shown in the Figure, 141 subjects were excluded due to having no available telephone number, and 2 were excluded due to having no mailing address. The remaining 12,285 subjects were randomly allocated to 1 of the 3 study arms, resulting in 4095 subjects per arm.

Subjects’ baseline demographic and clinical characteristics are summarized in Table 1. Asthma was the qualifying condition for 94% of the population, COPD for 5%, and asthma with COPD for 1%.

Information on completion of the reminder calls, for the IVR-only and the postcard-plus-IVR arms, was as follows: for 56.6% and 57.9%, respectively, messages were left on answering machines; 5.6% and 5.2%, respectively, listened to a vaccination reminder but hung up prior to receiving additional information; 20.4% and 19.3%, respectively, listened through to the message from the asthma/COPD specialist; 1.3% and 1.5%, respectively, listened to all content and requested more information from the influenza hotline; and 16% and 16% of calls, respectively, were incomplete with no answer and no message left.

By October 31, 2012, 29.5% of subjects in the postcard-only arm, 31.1% in the IVR-only arm, and 30.6% in the postcard-plus-IVR arm had received influenza vaccination. As shown in Table 2, no significant difference was found in vaccination rates among the 3 study arms for the primary outcome. Table 2 also shows influenza vaccination receipt by December 31, 2012, and March 31, 2013.

Multivariable analyses of factors associated with receipt of influenza vaccination are shown in Table 3. After adjusting for other covariates, vaccination was not significantly higher for the IVR-only and postcard-plus-IVR arms compared with the postcard-only arm. Of interest, several covariates were significantly associated with vaccination, which was significantly more likely in older age groups, for example. Compared with those with a traditional managed care insurance plan, those with a deductible plan were significantly less likely to receive influenza vaccine (adjusted odds ratio, 0.89; 95% CI, 0.84-0.94).

The cost associated with the different types of reminders is shown in Table 4. In the postcard-only arm, the cost per subject to receive the intervention was $0.78; for IVR-only, the cost was $1.23; and for postcard-plus-IVR, the cost was $1.93. When these costs were extrapolated to a theoretical population of 100,000 subjects, the anticipated costs per subject would be $0.55, $0.05, and $0.60 in the postcard-only, IVR-only, and postcard-plus-IVR groups, respectively. We performed an additional analysis to find the “break-even” point; 7916 postcards could be sent for the same cost as 100,000 IVR calls.


In this investigation of different reminder methods for influenza vaccination among adults with asthma or COPD, using IVR-based reminders was not significantly more or less effective for promoting influenza vaccination compared with postcard reminders or postcard-plus-IVR reminders. In addition, the IVR calls were not as interactive or tailored as was intended, as only 20% listened to a message from an asthma/COPD specialist, and only 2% requested information from the influenza hotline. The intervention achieved only modest influenza vaccination rates of 45% to 47% by the end of the influenza season, and the strongest predictor of vaccination was receipt of influenza vaccine in prior years. In cost analyses in the study population of 12,285 individuals, IVR reminders were more costly than postcard reminders due to the higher fixed costs of the IVR system. However, when costs were extrapolated to the entire high-risk population at KPCO, IVR-based reminders would have been the least expensive among the methods tested. These findings led KPCO to use IVR-based reminders as the primary method for influenza vaccination reminders for the entire high-risk population at KPCO for the 2013-2014 influenza season.

Although it was anticipated that IVR reminders would be more effective than postcard reminders for encouraging influenza vaccination, vaccination rates by October 31, 2012, were similar across the 3 study arms. IVR is considered a promising tool for health promotion and disease management because messages can be personalized and respondents can interact with the system through voice or touch-tone responses via telephone.25-27 However, in this study, use of the IVR system did not result in substantial interaction with respondents, because most either had a message left on an answering machine or did not choose to receive additional influenza information through the hotline. It is possible that higher immunization rates could have been achieved had more individuals listened to the additional information provided.

Nationally, during the 2012-2013 influenza season, an estimated 47% of adults aged 19 to 64 years with chronic medical conditions were immunized against influenza,8 and 34.6% of adults with asthma aged 18 to 49 years were immunized.28 These influenza immunization rates are considerably lower than the national goal of 90% coverage.9 The current study was conducted among insured adults in a managed care organization with walk-in immunization available—a setting presumably with fewer barriers to vaccination than faced by the general population. However, across the 3 study arms, only 45% to 47% were immunized by the end of the influenza season; several factors may have contributed to the lower-than-expected rates. Because KPCO patients are sent reminders every year and are potentially exposed to other types of reminders within KPCO (eg, in Kaiser newsletters), awareness may be high enough that patients do not need the additional cue to action that reminders provide. Additionally, it is possible some subjects were vaccinated outside of KPCO, and that this information was not entered into the KPCO EHR, resulting in falsely low rates.29 Finally, misperceptions about the need for influenza vaccination are prevalent, even among individuals with chronic medical conditions30,31; it is possible that the brief reminders used in the current study did little to change attitudes and behaviors.

While annual influenza vaccination reminders have proven effective in a variety of settings when compared with no reminders, much less is known about the cost of various reminder methods. Interestingly, IVR-based reminders were more costly than postcards for the study population of 12,285, but substantially less costly than postcards when applied to a much larger population, a finding that led KPCO to adopt IVR-based reminders for the entire high-risk population for the subsequent influenza season.


This study is subject to several important limitations. The study did not include a control group that received no reminders; however, because vaccination reminders are a recommended standard of care nationally, and a long-standing part of usual care at KPCO, having a “no reminder” study arm was not appropriate on ethical grounds. Reminders were left on answering machines, but it is not known whether these messages were ultimately heard by the intended recipients. Some subjects may have received influenza vaccination outside of KPCO, such as at a pharmacy or workplace, and this information would not routinely be captured within the EHR.29 While standard cost-capture methods were used, it is possible that not all reminder costs were measured. Finally, IVR systems can be expensive, costing $50,000 or more to initially implement or purchase. The IVR purchase or start-up costs were not included in our IVR reminder cost estimates because the KPCO IVR system has been in place for more than a decade, it is currently used for multiple purposes, and it is unlikely that a healthcare entity would purchase an IVR system solely for influenza vaccination reminders. This may limit the generalizability of our cost findings to organizations with existing IVR systems or those willing to purchase one, and we are unaware of published estimates of how prevalent IVR systems are among healthcare organizations.


In summary, in a randomized control trial, IVR-based reminders were not more or less effective for promoting annual influenza vaccination than postcard reminders. While IVR is a promising new technology for health promotion, when IVR-based reminders were used for influenza vaccination, many patients did not receive any tailored messaging as intended. Regardless of the type of reminder used in this study, only modest influenza vaccination rates were achieved, indicating that additional strategies may be needed to overcome misperceptions about the need for annual influenza vaccination.


We gratefully acknowledge Jeff Holzman for data management and Jonah Langer and Jenn Boggs for their technical assistance with the IVR delivery system.

Author Affiliations: Institute for Health Research (JAS, JE, DPR, MFD) and Department of Population and Prevention Services (CM, CK, CL), Kaiser Permanente Colorado, Denver, CO; Department of Pediatrics, University of Colorado School of Medicine (MFD), Aurora, CO.

Source of Funding: This study was funded by an internal pilot grant from Kaiser Permanente Colorado’s Institute for Health Research.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. Trial registration: NCT01852656, Effectiveness of Influenza Vaccine Reminder Systems.

Authorship Information: Concept and design (JAS, DPR, MFD, CK, CM, CL); acquisition of data (JAS, JE, MFD, CK, CM, CL); analysis and interpretation of data (JAS, DPR, JE, MFD, CK, CM); drafting of the manuscript (JAS, DPR, JE, MFD, CL); critical revision of the manuscript for important intellectual content (JAS, DPR, JE, MFD, CM); statistical analysis (JAS, JE); obtaining funding (MFD); administrative, technical, or logistic support (JAS); supervision (MFD).

Address correspondence to: Jo Ann Shoup, MS, Institute for Health Research, Kaiser Permanente Colorado, 10065 E Harvard Ave, #300, Denver, CO 80231. E-mail:

1. Rothberg MB, Haessler SD. Complications of seasonal and pandemic influenza. Crit Care Med. 2010;38(4 suppl):e91-e97. Review.

2. Glezen WP, Greenberg SB, Atmar RL, Piedra PA, Couch RB. Impact of respiratory virus infections on persons with chronic underlying conditions. JAMA. 2000;283(4):499-505.

3. Griffin MR, Coffey CS, Neuzil KM, Mitchel EF Jr, Wright PF, Edwards KM. Winter viruses: influenza- and respiratory syncytial virus-related morbidity in chronic lung disease. Arch Intern Med.
4. Tan WC, Xiang X, Qiu D, Ng TP, Lam SF, Hegele RG. Epidemiology of respiratory viruses in patients hospitalized with near-fatal asthma, acute exacerbations of asthma, or chronic obstructive pulmonary disease. Am J Med. 2003;115(4):272-277.

5. Skrepnek GH, Skrepnek SV. Epidemiology, clinical and economic burden, and natural history of chronic obstructive pulmonary disease and asthma. Am J Manag Care. 2004;10(5 suppl):S129-S138. Review.
6. CDC. Prevention and control of influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP)—United States, 2012-13 influenza season. MMWR Morb Mortal Wkly Rep. 2012;61(32):613-618.
7. Fiore AE, Uyeki TM, Broder K, et al; CDC. Prevention and control of influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP), 2010. MMWR Recomm Rep2010;59(RR-8):1-62.
8. National Immunization Survey and Behavioral Risk Factor Surveillance System [data sources]. Flu vaccination coverage, United States, 2012-2013 influenza season. CDC website. htm#key-findings 2013. Updated September 25, 2013. Accessed October 22, 2013.
9. Healthy People 2010: Understanding and Improving Health. 2nd ed. Washington, DC: HHS; 2000. 

10. Jacobson Vann JC, Szilagyi P. Patient reminder and patient recall systems to improve immunization rates. Cochrane Database Syst Rev2005;(3):CD003941.
11. Thomas RE, Russell M, Lorenzetti D. Interventions to increase influenza vaccination rates of those 60 years and older in the community. Cochrane Database Syst Rev. 2010; (9):CD005188. Review.
12. Lieu TA, Capra AM, Makol J, Black SB, Shinefield HR. Effectiveness and cost-effectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization. Pediatrics. 1998;101(4):E3.

13. Suh CA, Saville A, Daley MF, et al. Effectiveness and net cost of reminder/recall for adolescent immunizations. Pediatrics2012;129(6):e1437-e1445.
14. Kempe A, Saville A, Dickinson LM, et al. Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial. Am J Public Health.
15. Corkrey R, Parkinson L. Interactive voice response: review of studies 1989-2000. Behav Res Methods Instrum Comput2002;34(3):342-353.
16. Estabrooks PA, Smith-Ray RL. Piloting a behavioral intervention delivered through interactive voice response telephone messages to promote weight loss in a pre-diabetic population. Patient Educ Couns2008;72(1):34-41.

17. Estabrooks PA, Shoup JA, Gattshall M, Dandamudi P, Shetterly S, Xu S. Automated telephone counseling for parents of overweight children: a randomized controlled trial. Am J Prev Med. 2009;36(1):35-42.

18. Skolarus TA, Holmes-Rovner M, Hawley ST, et al. Monitoring quality of life among prostate cancer survivors: the feasibility of automated telephone assessment. Urology. 2012;80(5):1021-1026.

19. Von Korff M, Wagner EH, Saunders K. A chronic disease score from automated pharmacy data. J Clin Epidemiol. 1992;45(2):197-203. 

20. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373-383.

21. Neter J, Kutner MH, Nachtsheim CJ, Wasserman W. Applied Linear Statistical Models. 4th ed. Boston, MA: WCB/McGraw-Hill; 1996.

22. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide to enhance translation. Ann Behav Med. 2009;37(2):218-227.

23. Ritzwoller DP, Sukhanova AY, Glasgow RE, et al. Intervention costs and cost-effectiveness for a multiple-risk-factor diabetes self-management trial for Latinas: economic analysis of ¡Viva Bien! Transl Behav Med. 2011;1(3):427-435.

24. Ritzwoller DP, Glasgow RE, Sukhanova AY, et al; Be Fit Be Well Study Investigators. Economic analyses of the Be Fit Be Well program: a weight loss program for community health centers. J Gen Intern Med. 2013;28(12):1581-1588.
25. Piette JD. Interactive voice response systems in the diagnosis and management of chronic disease. Am J Manag Care. 2000;6(7):817-827. 

26. Stacy JN, Schwartz SM, Ershoff D, Shreve MS. Incorporating tailored interactive patient solutions using interactive voice response technology to improve statin adherence: results of a randomized
clinical trial in a managed care setting. Popul Health Manag2009;12(5):241-254.
27. Noar SM, Harrington NG, eds. eHealth Applications: Promising Strategies for Behavior Change. New York, NY: Routledge; 2012. 

28. CDC. Vaccination coverage among persons with asthma—United States, 2010-2011 influenza season. MMWR Morb Mortal Wkly Rep2013;62(48):973-978.
29. Greene SK, Shi P, Dutta-Linn MM, et al. Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med. 2009;37(6):552-555.

30. Nichol KL, Lofgren RP, Gapinski J. Influenza vaccination: knowledge, attitudes, and behavior among high-risk outpatients. Arch Intern Med. 1992;152(1):106-110.
31. Kempe A, Daley MF, Crane LA, et al. Misperceptions regarding influenza vaccine safety for individuals with chronic medical illness. Prev Med. 2007;45(1):80-82.
Copyright AJMC 2006-2018 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up

Sign In

Not a member? Sign up now!