Sheri A. Keitz, MD, PhD; David C. Aron, MD; Judy L. Brannen, MD; John M. Byrne, DO; Grant W. Cannon, MD; Christopher T. Clarke, PhD; Stuart C. Gilman, MD; Debbie L. Hettler, OD, MPH; Catherine P. Kaminetzky, MD, MPH; Robert A. Zeiss, PhD; David S. Bernett, BA; Annie B. Wicker, BS; and T. Michael Kashner, PhD, JD
As healthcare systems consider investments in clinical education, they desire evidence to support whether healthcare training provides a return on investment. Evidence exists that successful recruitment and retention of healthcare professionals is associated with higher employee satisfaction,1
which in turn is related to improved patient satisfaction2
and quality of care.3
However, there is a paucity of evidence quantifying any association between training and willingness of trainees to work in the systems where they trained. The Department of Veterans Affairs (VA) healthcare system, with long-standing investments in healthcare training, provides a model to study this association that is applicable to broader settings. The foundation for this model is based on evidence that clinical training experiences and program factors have been shown4-9
(also J.M.B. et al, unpublished data, 2017) to be related in the VA.
In the United States, healthcare trainees’ career choices regarding location and specialty are not strategically aligned with healthcare needs, especially in primary care, rural areas, and underserved populations that include veterans.10
Traditional models to explain location and specialty choices among physicians have emphasized trainee attitudes and sociodemographic and environmental factors.11
Previous studies have focused on predictors including debt load, earning potential,12
lifestyle, and specialty prestige.13
The location of physician training programs has also been reported as a driver of career choice, as residents often elect to stay in the geographic areas where they trained,14
including in small towns,15
and rural settings.17
Furthermore, rural hospitals with residency programs were more successful at recruiting and retaining physicians than rural hospitals without residency programs.18,19
Few studies have examined career location and practice choices among nonphysician health professionals, critically including those in nursing and associated health (AH) disciplines (see Table 1 [part A and part B]
for definition of AH).
We examine 2 questions pertaining to whether a training experience is sufficient to critically affect career choices across healthcare disciplines. Did the clinical learning experiences of physician, nursing, and AH profession trainees
influence their consideration about future employment in VA? If yes, what were the salient factors that led to those changes in perceptions?
The study took place in 169 VA facilities in which health professions students, interns, residents, fellows, and other trainees rotate as part of an education curriculum under affiliation agreements (38 USC 7302) between VA and 135 allopathic medical schools, 30 osteopathic medical schools, and more than 1800 colleges and universities.
Data and Design
This was a pre–post intervention study in which subjects served as their own controls. Observational data came from a convenience sample of trainees who had concluded their rotation through a VA medical center between July 1, 2014, and June 30, 2016, and who agreed to take the survey, entered a designated website, and completed the survey. Overall, 85% of respondents who entered the website completed the survey. All VA trainees were invited to enter the website and complete the survey through emails and direct contacts by VA program officers at the time the trainee exited from VA.
At the end of their rotation, respondents were administered the Learners’ Perceptions Survey (LPS),20
an anonymous, voluntary, online survey. Here, respondents were asked about their perceptions of VA covering 3 time periods: prerotation, during their clinical rotation, and immediately after their rotation. The 1-survey administration design allows us to match pre–post perceptions without having to identify trainees, who are considered to be vulnerable human subjects.
Since 2000, VA’s Office of Academic Affiliations has administered the LPS under the Office of Management and Budget’s Information Collection Authorization (#2900-0691, VA form #10-0439) within a confidential Data Accounting System.21
LPS response data used for this study show high levels of content validity, test–retest reliability, scalability, internal consistency, and discriminant and congruent construct validity.5,22,23
LPS physician respondents are comparable with residents (excluding pediatrics and obstetrics/gynecology [OB/GYN]) in US accredited graduate medical education programs in 20134
and 2015 with respect to gender, specialty, international graduate status, and postgraduate year (PGY) (J.M.B. et al, unpublished data, 2017).
Our purpose is to measure the change in a trainee’s willingness to consider VA for future employment following their VA training experience, here referred to as recruitability. Pretraining perception is determined by asking, “Before this VA clinical training experience, how likely were you to consider a future employment opportunity at a VA medical facility?” and posttraining perception by asking, “As a result of this VA clinical training experience, how likely would you be to consider a future employment opportunity at a VA medical facility?” on 5-point ordinal scales with response choices of very unlikely, somewhat unlikely, had not thought about it, somewhat likely, and very likely.
The LPS also asks respondents to rate their clinical training during their VA rotation on 6 satisfaction and 3 assessment domains. Satisfaction was rated on a 5-point scale for the 7-item clinical environment, 9-item working environment, 8-item physical environment, 7-item personal experience, 15-item learning environment, and 13-item clinical faculty and preceptors domains. To compute satisfaction domain scores, a value of 1 was assigned to responses of very dissatisfied, 2 to somewhat dissatisfied, 3 to neither satisfied nor dissatisfied, 4 to somewhat satisfied, and 5 to very satisfied. Domains were scored as Likert scales24
by taking the mean response across all items within each domain. This is appropriate because the items have been shown to constitute a single dimension.22,23
Assessment domains were computed based on how much respondents agreed or disagreed with a statement describing their training experiences. Items were scored by assigning a value of 1 to responses of strongly disagree, 2 to disagree, 3 to neither disagree nor agree, 4 to agree, and 5 to strongly agree. As with satisfaction, assessments were scored as the mean responses across n
items for 2-item psychological safety, 16-item patient-centered care, and 9-item interprofessional team care domains.
LPS responses were also used to assess other covariates. Facility-level factors include complexity as scored officially by VA on a 5-level scale based on number of beds, clinical services offered, number and diversity of trainees, and dollars of research support.25
VA also computes a rurality index as the ratio of a facility’s census of rural and highly rural patients to total patient census. At the respondent level, factors are the trainee’s gender, length of time in VA, academic program, and academic level. The mix of patients seen during their VA rotations is computed across 8 variables. These variables describe the percent of patients the respondent saw who were 65 years or older; female; with a chronic mental illness, a chronic medical illness, multiple medical illnesses, and alcohol/substance dependence; with low income/socioeconomic status; or lacking social/family support. To avoid confounding with professional discipline, patient mix factors were centered around the mean for all respondents with the same professional discipline, specialty, and academic level. Finally, as described elsewhere,8,22
associations were calibrated to reflect differences in how trainees apply thresholds when classifying the intensity of their satisfaction into 1 of the 5 response levels. This calibration index is computed by taking the respondent’s mean satisfaction rating for facility-level items (facility convenience, parking, and electronic health record) and subtracting the average of these mean ratings across all respondents at the respondent’s VA facility and reporting year.
Recruitability, both before and after training, was measured on a 5-point ordinal scale and treated analytically as repeated measures, based on generalized estimating equations with a multinomial distribution and cumulative logit link function. The training effect was computed as an odds ratio by discipline based on estimates of the coefficient to an independent time indicator variable that equals 1 if post VA training and 0 if pre–VA training. Factors related to these associations were estimated using interaction terms calculated by multiplying a suspected mediator with the time indicator variable. Academic level (years of training) was centered to reflect a referent academic level that describes the level at which a trainee was most likely to be exposed to VA for the first time in that discipline (eg, third-year medical student, PGY1 physician resident, PGY4 resident in medicine subspecialty; see Table 2
). Finally, the mean-centered patient mix and calibration index variables were included as control variables to reduce confounding in estimated associations between training experience and recruitability. For this exploratory study, a Wald χ2
test statistic is computed, and 95% CIs and P
values provided, to indicate precision of effect size estimates, and not for hypothesis testing.
Table 1 describes characteristics of survey respondents; they represent a broad range of professional disciplines, subspecialties, academic levels, gender, mix of patients seen, facility location (rural or urban), complexity of care, and respondents’ perceptions of training experiences across 9 domains. The domains are presented in descending rank of satisfaction or agreement for all trainees. Overall, physician trainees tended to rate their clinical training experiences lower than their nursing or AH counterparts. Faculty/preceptors was the highest-rated and interprofessional team care was the lowest-rated domain across professions.
Overall Pre–Post Training Effect on Recruitability by Discipline
As shown in Table 2, VA trainees were 2.1 times more likely to consider VA employment after VA experiences, with recruitability increasing from 1.6-fold for physicians to 2.5-fold for nursing and 2.7-fold for AH trainees. The largest before–after differences were reported for blind and other rehabilitation therapy (4.5), speech pathology (4.3), and occupational (4.2) and physical (4.1) therapy. Podiatry residents were the only profession reporting no net increase in recruitability following VA training.
Impact of Clinical Experience and Program Factors on Recruitability
reports how satisfaction and assessment ratings of the clinical experiences and program factors were associated with pre–post changes in recruitability for physician, nursing, and AH profession trainees. For all 3 groups, higher ratings for learning environment, personal experiences, and psychological safety were associated with larger changes in recruitability. Physician trainees were also influenced by how they rated their clinical environment, whereas nursing students and AH trainees were influenced by how they rated faculty/preceptors. In contrast, the presence of interprofessional team experiences and patient-centered care were unrelated to changes in recruitability. In fact, clinical environments emphasizing interprofessional team care were associated with smaller changes in recruitability for nursing students. Ratings of satisfaction with physical environment did not influence the impact of training on recruitability for any group. Among training program characteristics, training in more complex facilities had a slight advantage in increasing recruitability. Gender mattered only for AH, where female trainees were observed to be 1.2 times more likely to change perceptions about VA employment than were their male counterparts.
Prior to training, trainees at rural VA medical centers were only 59.6% (95% CI, 52.2%-67.9%; P
<.001) as likely to consider VA employment as their urban counterparts. VA training narrowed this gap, but only for AH trainees. That is, AH trainees at rural VA medical centers were initially only 41.9% (95% CI, 33.8%-51.9%; P
<.001) as likely to consider VA employment as their urban counterparts. After VA training, this increased by an adjusted factor of 1.77 (95% CI, 1.44-2.17; P
<.001) to 74.1% (95% CI, 58.0%-94.6%; P
VA medical centers face many of the same challenges of staff recruitment and retention as academic medical centers in the private sector that engage in patient care, health profession education, and research. This study found that clinical trainees from all disciplines, except podiatry, were substantially more likely to consider VA for future professional employment after clinical training at a VA facility than before. This pre–post increase in recruitability was associated not only with structural factors, but also with their clinical training experiences. This is consistent with the theory that mere exposure to a teaching facility is not sufficient to induce trainees to change their perceptions about whether they will want to work there in the future. These data suggest that across disciplines, urban/rural settings, and facility complexities, trainees must give high ratings to their learning environment, personal experiences, and sense of psychological safety before their VA training experiences are expected to be associated with improved willingness to consider VA for future employment.
Our findings provide a business case to support VA’s long-standing affiliations with the nation’s academic institutions.26,27
VA Policy Memorandum Number 2, published January 30, 1946, codified relationships between VA facilities and medical schools that gave veterans the highest quality medical care and opportunities to further their education, raise the standard of medical practice in the United States, and mitigate the shortage of physicians after World War II.28,29
These data provide evidence that the network of academic partnerships increased healthcare trainee willingness to work in VA14
and are consistent with the policy behind the Veterans Access, Choice, and Accountability Act of 2014 [PL 113-146; 38 USC 7302(e) (1)-(2)], which expands the number of funded VA physician resident and fellow positions in primary care, mental health, and other specialties by 1500 over 10 years.30,31
Psychological safety is defined as a sense of safety in taking personal risks to discuss problems without fear of retribution. Our data suggest that psychological safety is associated with future employment choices across all trainee disciplines. Our finding is consistent with earlier reported positive associations between psychological safety and trainee satisfaction with the clinical learning environment,8
error reporting, team performance, personal work engagement and systems improvement,32-34
and mental health employees’ job satisfaction and turnover intentions.35
Collectively, these studies underscore the role that psychological safety plays in the quality of clinical training programs.
Personal experiences include job stress, fatigue, and capacity to balance personal and professional life. Our data showing that personal experience is important to future employment preferences of physician, nursing, and AH profession trainees are consistent with prior findings that controllable lifestyles influence specialty choices.36,37
While lifestyle factors have also been shown to be important to dental students,38
the literature does not offer insight into the effect of lifestyle or personal experiences on employment choice of other AH trainees.
The clinical faculty/preceptors domain includes faculty clinical skills, teaching ability, quality, being evidence-based, mentoring, serving as a role model, and accessibility and approachability. Our finding—that receptiveness to a career in VA was associated with how nursing and AH profession trainees rated their clinical faculty/preceptors—is consistent with the importance these programs place on learning through student interactions with faculty and preceptor role models.39
Our findings that physician preferences for future employment were unresponsive to faculty/preceptors are in contrast to those of other studies that showed that faculty and preceptors affected physician career specialty choices.40
These data failed to show a direct link between a trainee’s recruitability and VA experience with interprofessional team or patient-centered care. In fact, the presence of interprofessional team care for nursing students and patient-centered care for physician trainees had a negative influence on openness to consider VA for future employment. This is important. Both VA care models and academic institutions’ curricula emphasize patient-centered and interprofessional team care. For example, surgical team training in VA has been associated with improved mortality. Our findings are consistent with those of earlier studies that showed that physician trainees undervalue patient-centered and interprofessional team care compared with nonphysician trainees, even after changes in medical school and residency training curricula.41
On the other hand, they contrast with a finding that internal medicine residents rotating through VA primary care continuity clinics are more likely to value patient-centered and interprofessional team care than colleagues at non-VA continuity clinics.42
More studies are needed to understand the relationship between how trainees value patient-centered and interprofessional team care and their future employment choices.
Our finding that female AH trainees were more responsive to recruitability following training than male AH trainees is consistent with the psychology literature that suggests differential influences on job choices between women and men. One study showed that more on-the-job engagement is associated with decreased job turnover and that this association was stronger in female-dominated samples.43
These findings also shed light on the critical priority of enhancing recruitment of health professionals in rural areas. Overall, trainees in rural facilities began and ended their training with a poorer view of VA employment than their urban counterparts. However, rurality was a positive modifier for AH trainees, who demonstrated a 1.77-fold increase in recruitability (Table 3). This underscores the critical importance of establishing quality health professions training programs in rural settings. An important caveat is that an increase in recruitability does not necessarily imply increased willingness to work for VA in a rural versus urban setting.
This study has limitations. First, this is a convenience sample subject to selection biases. Despite adjusting estimates to account for mediating interactions and blinding participants to the study purpose, some selection biases will likely remain. In prior studies,4
(also J.M.B. et al, unpublished data, 2017) LPS respondents who were physician residents were comparable by gender, academic level, international status, and specialty with residents in all accredited, nonpediatric, and non-OB/GYN US residency programs. Second, our use of observational data will leave estimates of associations subject to confounding biases. However, randomized trials are not practical here.44
Mitigating this bias was a pre–post comparison in which subjects serve as their own controls, and final estimates are adjusted for respondent and facility factors and calculated by discipline. Third, both pre- and posttraining recruitability questions were administered in the same survey, introducing recall bias and possibly blurring differences to create downward biases on the estimates of associations between training and recruitability. Fourth, the data are not linked to actual employment decisions made by respondents. However, intentions to stay have been shown to be related to job search behavior, job performance, and, ultimately, actual turnover.35
Future studies should follow respondents and their actual employment decisions. Fifth, cross-profession comparisons may be biased by how healthcare professionals in different disciplines perceive satisfaction.45
To mitigate these biases, estimates were adjusted to reflect variation in response thresholds to account for how respondents approached classifying the intensity of their perceptions into 5-level ordered responses. Finally, variation in organizational factors may also affect the trainee’s employment choices. However, there is little evidence that VA reorganization in specialties such as psychiatry had meaningful impacts on employee satisfaction or perceptions of their work environment.46
To recruit professional staff from among their physician, nursing, and associated health trainees, academic medical centers are advised to invest in their training mission in order to improve trainee clinical, working, learning, and cultural experiences.
The authors express their sincere gratitude for the support from the Office of Academic Affiliations (OAA) (Jemma Ayvazian, DNP, ANP-BC, AOCNP, and Samuel S. King, MS, MDiv), OAA’s National Evaluation Workgroup (Sheri A. Keitz, MD, PhD, chair), OAA Data Management and Support Center in St. Louis, Missouri (Christopher T. Clarke, PhD; David Bernett, BA; Terry V. Kruzan; George E. McKay; and Laura Stefanowycz), and the network of Designated Education Officers and Associated Chiefs of Staff for Education at VA Medical Centers.