Paul A. Heidenreich, MD, MS; Aimee Zapata, MS; Lisa Shieh, MD, PhD; Nancy Oliva, PhD, RN; and Anju Sahay, PhD
Recently, the Department of Veterans Affairs (VA) started the Choice Program (section 101 of the Veterans Access, Choice, and Accountability Act of 2014), which pays for veterans to receive care at non-VA hospitals if care cannot be provided in a timely manner (within 30 days) at a VA hospital or if the patient lives a significant distance (greater than 40 miles) from one.1
Hospital Compare, a website maintained by CMS, allows comparisons of outcome and processes of care for VA and non-VA hospitals.2
Patient satisfaction is reported on Hospital Compare for non-VA hospitals only. Currently, the only way for patients to compare VA and non-VA hospitals on patient experience is through private online customer rating services (eg, Yelp).
A recent study found a high correlation between Yelp ratings and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores as reported by Medicare on Hospital Compare for non-VA US hospitals.3
Another comparison of HCAHPS and Yelp ratings for non-VA US hospitals found that Yelp ratings covered the main domains of HCAHPS while providing information on additional domains of patient experience.4
These data suggest that Yelp may be a reasonable alternative for estimates of the patient experience at VA hospitals.
A common alternative to a VA hospital is an affiliated university hospital. Over 90% of VA hospitals are affiliated with medical schools and many share trainees and physician staff with university hospitals. We sought to compare online hospital ratings between VA and local affiliated hospitals. This allowed us to compare the patient experience at VA and non-VA hospitals in the same region that often share trainees and physicians.
We searched Yelp for hospital ratings for the 131 VA hospitals from March 2014 to June 2014 and recorded the Yelp ratings 1 (worst) to 5 (best). Ratings were excluded if the category was specific to a department (eg, cafeteria). We only included hospitals if there were 2 or more online ratings. Therefore, we attempted to identify the local university hospital affiliate of each of the VA hospitals. If there was more than 1 local affiliate, we used the affiliate with the greatest number of online reviews. For 1 VA hospital for which there was no local affiliate, we selected a random hospital within 5 miles that had online reviews. We determined hospital characteristics (bed size, membership in the Council on Teaching Hospitals [COTH], presence of an accredited graduate medical education program, and certification by The Joint Commission [TJC]) using data from the American Hospital Association.
Primary Measure of Patient Satisfaction
We determined the mean Yelp rating for each hospital.
Continuous variables of hospital characteristics were compared between VA and non-VA hospitals using t
tests. Categorical variables of hospital characteristics were compared using χ2
tests. We first compared the mean Yelp ratings of the VA and affiliated hospitals using an unweighted paired t
test. In a sensitivity analysis, comparisons of mean ratings were weighted by the number of ratings using analysis of variance. The Shapiro-Wilk test did not reject the assumption of normality for the rating distribution (P
= .94). We then examined the correlation (clustering) of online ratings for the pairs of VA and non-VA hospitals. Finding negligible clustering (see results for correlation), we used ordinary least squares multivariate regression to compare mean Yelp ratings, adjusting for hospital characteristics (bed size, membership in COTH, presence of an accredited graduate medical education program, and certification by TJC). We repeated the analyses limiting the sample to those with 5 or more, 10 or more, and 15 or more individual ratings. A 2-sided P
<.05 was considered statistically significant. All analyses were performed with Stata version 11.0 (Stata Corp; College Station, Texas).
We identified 39 VA hospitals (30%) and 39 non–VA-affiliated hospitals with at least 2 Yelp ratings. Hospital characteristics are displayed in the Table
. Non-VA hospitals were larger than VA hospitals. The vast majority of both VA and non-VA hospitals were affiliated with medical schools, although non-VA hospitals were more likely to be a member of COTH. Almost all used TJC for hospital accreditation. VA hospitals with less than 2 Yelp reviews (excluded from the study, n = 92) were smaller and less likely to be academic than those VA hospitals with 2 or more ratings (eAppendix Table
[eAppendices available at ajmc.com]
The mean patient rating (± standard deviation) for VA hospitals was higher (3.64 ± 1.0) than the rating for affiliated hospitals (3.09 ± 0.75; P
= .0036, t
test-unweighted). The median difference between VA and affiliate ratings (VA – non-VA) was 0.5 (interquartile range, –0.5 to 1.5). After weighting for the number of reviews (analysis of variance), the mean patient ratings were slightly higher for both VA hospitals (3.70 ± 0.74) and affiliated hospitals (3.19 ± 0.54), although the difference remained similar and significant (P
= .0025). This difference in online ratings persisted when the sample was limited to dyads of hospitals (VA and non-VA) with more than 2 reviews (Figure
). There was no significant correlation in rating between a VA hospital and its affiliate (r = 0.07; P
= .59). After adjustment for hospital characteristics, the adjusted rating difference (VA-affiliate hospital) was 0.65 (95% confidence interval, 0.18-1.12).
Patient satisfaction with care is increasingly recognized as important by payers, including CMS. Hospital ratings on the HCAPHS surveys are publically reported2
and used as part of pay-for-performance programs. Patient-initiated reviews have limitations, but a recent study found a strong correlation (r = 0.49; P
<.0001) between Yelp reviews and HCAHPS scores for non-VA US hospitals.3
At the time of the study (2011), 25% of all non-VA US hospitals had a Yelp review, and this percentage increased over time—this percentage is consistent with our ability to find reviews for 30% of VA hospitals. The mean score for all non-VA US hospitals was 3.3 in 2011, slightly higher than the non-VA hospital score in our study.
Although Yelp reviews describe patient experience, they may also be a marker of patient outcome. Our study was not powered to detect differences in outcome; however, the Bardach study3
found that high Yelp ratings were significantly associated with lower 30-day all-cause mortality following an admission for myocardial infarction or pneumonia and lower 30-day all-cause readmissions following a discharge for myocardial infarction, heart failure, or pneumonia.
Our study is potentially limited by our comparison group, which included local university-affiliated hospitals that often share trainees and faculty with the VA hospital. Although veterans may often choose the local affiliate hospital to receive care, our results may not be applicable to VA comparisons with non-affiliated community hospitals. We excluded 70% of VA hospitals with fewer than 2 reviews and these were smaller and less academic than the included studies sites. A prior study found that smaller hospitals had higher patient satisfaction ratings than larger hospitals although there was no significant impact of teaching status on ratings.5
Differences in patients could explain the results. For example, we cannot know if veterans are more likely than nonveterans to rate all hospitals highly.
We found that VA hospitals have higher patient ratings than their affiliated hospitals, a finding not explained by bed size or teaching status.