
Do Wellness Outcomes Reports Systematically and Dramatically Overstate Savings?
When analyzing the effect of workplace wellness and related employee health services, studies invariable attribute savings among participants to the program rather than the more likely
The vast majority of “studies” in the field of workplace wellness and related employee health services compare participants to nonparticipants, and show substantial savings in the former vs the latter. They invariably attribute the savings among the participants to the program (a “program effect”) rather than to the likely much higher level of motivation among participants to succeed in any program (the “participation effect”).
For example, wellness promoters claim that if you divide a company into employees who want to lose weight vs employees who don’t, that the difference in weight loss between the 2 groups is due to the program, not the relative difference in motivation to lose weight. (And, further, that people who start in the motivated category but drop out shouldn’t count at all.)
If indeed it is the case that the “participation effect” is present and substantial, savings from all studies with that design are overstated. And yet despite its ubiquity in wellness study designs, no one has ever postulated the existence of a participation effect using data. This is particularly perplexing because, intuitively, it makes no sense that, in an undertaking such as personal health improvement in which motivation is key, separating a population into study and control groups on the basis of motivation would constitute a valid study design.
Unsurprisingly, it is easily provable that the intuitive result is indeed the correct result.
How do we know this? There have been 3 studies in which the participation effect can be isolated from the program effect. Further, each study featured the opposite of “investigator bias,” in that the authors were attempting to prove (and, indeed, thought they had proven) the opposite.
Each study demonstrated the substantial impact of the participation effect from different angles:
- Case Study 1: Would-be participants were separated from nonparticipants but not offered a program in which to participate … and yet showed massive savings;
- Case Study 2: A program gave what is now widely believed to be incorrect advice to participants, the advice was taken, and risk factors barely fell … but participants still showed massive savings vs nonparticipants;
- Case Study 3: A controlled experiment measured results both population-wide and by separating participants vs nonparticipants within that population. The population’s health was unaffected by the intervention, but participants showed massive savings vs non-participants within the population.
Case Study 1
The slide below—the
While this study would seem to constitute an excellent demonstration of the “participation effect,” there is one limitation: 4 years after the study was published, reviewed, and blessed by the Koop Award Committee,
That revision raises another question, though: by the end of 2008, the “savings” for Eastman participants exceeded $900/year, or 24%, but average
Even if one accepts that—despite its review by the entire Koop Award Committee and its subsequent wide dissemination—the key display was wrong the entire time and no one noticed until it was
Case Study 2
By encouraging participating diabetics and people at risk for diabetes to (among other things) eat less fat and, hence, likely more carbohydrates, Stanford University researchers showed a relative reduction in costs of $397 in the first 6 months alone vs nonparticipants. (Because this study was conducted in the 1990s, it modeled only $1000/day as the hospital per diem. The same claimed reduction in hospital utilization multiplied by a more current per diem would generate a far greater dollar savings.)
Today, recommending that people at high risk for diabetes eat less fat would be considered controversial, and no one would attribute substantial near-term savings to that recommendation. In general, it is also highly unlikely that substantial savings could be achieved within the first 6 months with any intervention reliant on long-term education and behavior change.
Further, and unsurprisingly, behaviors hardly changed anyway. Risk scores improved by only 2% vs control. Even that 2% might be an overstatement: the control group itself was composed of people with much lower risk scores than the study group and it was assumed that there was no regression to the mean in the high-risk group. However,
Since participant risk factors declined only 2% (vs. the control) likely due largely to regression to the mean, and since much of the advice was arguably wrong, one should conclude that the massive savings in the first 6 months—probably exceeding $1000 in today’s dollars—can be attributed to the participation effect rather than the dietary and other changes by 2% of the study group.
Case Study 3
This was a
Nor should there have been any difference. Subjects were Invited specifically because they were not chronically ill. Hence there was no opportunity for improve health status in a clinically meaningful way, especially in only 12 months. Further, the intervention was based on the highly questionable proposition that telling people they had a gene for obesity would motivate them to lose weight.
Let’s focus now just on the 2 Invited subsets—the participants and the nonparticipants.
Despite the overall result showing no separation between Control and Invited, the participants within Invited incurred $1464 less cost than the nonparticipants in Invited during the first year alone. This industry-record alleged savings was achieved by a DNA-based intervention whose efficacy is questioned by
Hence, the $1464 in savings is purely the participation effect at work, a completely compelling “natural experiment.”
Cases Indicate a Pattern in Wellness
These are just the 3 most self-evident cases. But there is a pattern in wellness,
The implication of these cases and this pattern? Listing “unobservable differences” between participants and nonparticipants as a limitation is no longer a sufficient disclaimer when reporting or publishing favorable results of wellness studies in which the former uses the latter as a control. A better description would be “invalidator.” At a minimum, investigators should point readers to this posting and suggest that they draw their own conclusions about validity.
Newsletter
Stay ahead of policy, cost, and value—subscribe to AJMC for expert insights at the intersection of clinical care and health economics.