
NBER's Invalidation of Wellness: Behind the Headlines
A new study represents an existential threat to conventional wellness programs. Here's a behind-the-scenes look at the study's findings.
Some tourist attractions feature an “A” tour for newbies and then a “behind-the-scenes” tour for those of us who truly need lives. For instance, I confess to having taken Disney’s Magic Kingdom underground tour, exploring, among other things, the tunnels through which employees travel so as not to be seen out of costume in the wrong “Land.”
Likewise, there have been many reviews of the
By way of background, the headline is that the mainstream wellness program the investigators examined at the University of Illinois did not noticeably move the needle on employee health. They didn’t address return-on-investment (ROI), because there obviously was none. Achieving a positive ROI would require moving the health risk needle—not just by a little, but rather by enough to significantly improve the health of many employees. Then, since wellness-related events, such as heart attacks, would not otherwise have befallen these employees immediately, this improvement would have to be sustained over several years before there was a statistical chance of some events being avoided.
Finally, the magnitude of this improvement would have to be great enough to violate the rules of arithmetic, because it is not mathematically possible
This finding, therefore, represents an existential threat to conventional wellness programs.
No surprise, then, that while virtually all the reviews of this study have been positive, wellness apologists have tried to discredit it 2 ways. First, they would say: “Yes, but it’s only 1 year.” That is true, but there is no reason to expect a “hockey stick” improvement in Year 2, when in Year 1 employees didn’t even increase their visits to the gym in Year 1, which would have been the easiest behavior improvement.
Next, detractors would, ironically, say: “Everyone knows savings don’t start in the first year.” And yet the Koop Awards—the award given to the best-in-country wellness program by a group of wellness promoters that licensed the name of former Surgeon General C. Everett Koop, MD, (which
Detractors would also say: “This study was misdesigned because they lumped nonparticipants and participants together. Obviously, nonparticipants would have no interest in this program.” That, of course, is a feature of this study design, not a bug. This study design controls for motivation, in order to isolate the program effect from the self-selection bias, such as the bias evident in that very same award-winning study.
All these negative reviews took place in LinkedIn groups that are generally closed to the public, and also to wellness critics. Despite multiple requests to comment, the Health Enhancement Research Organization (HERO, the wellness industry trade association) has offered no rebuttal, response, or even acknowledgement of the existence of this study. Perhaps it fears that its comments will draw more attention to the original, not just from trade publications, but also from
The study could be criticized on 1 key point, a flaw that no detractor spotted. The 2 groups were compared on the basis of change in total cost, rather than (or in addition to) the change in the total number of wellness-sensitive medical events (WSMEs). The “gold standard” methodology of tallying WSMEs is described in my book,
The reason to focus on WSMEs is that being screened for heart disease and diabetes, providing information on one’s diet and exercise, and having access to a gym could not be expected to reduce total costs, but rather just the cost of avoided WSMEs. Wellness programs aren’t going to get “medical costs to decrease significantly for neoplasms and digestive systems [sic], and for blood and blood-forming organs,” though
i The separation of participants (green line) and non-participants (orange line) took place in 2004. The program started in 2006. By the time the program started, participants had already “saved” almost $400, without even having a program to participate in.
The behind-the-headlines tour of the NBER study yields the most important conclusion
The focus on the headlines has obscured the most important conclusion from this study, which is that the entire participant vs nonparticipant methodology is invalid. Until now, that methodology had been 1 of the 2 methodologies the industry uses to reliably “show savings” in virtually any circumstance.ii
Of the 2, the participant vs nonparticipant methodology is the only one that arguably needs scientific proof along with mathematical invalidation. The mathematical invalidation has already been
The best confirmation of any initial conclusion is an analysis reaching a similar or identical conclusion—but conducted using a completely different approach, by completely different researchers, with no relationship to the original researchers, and no confirmation or investigation bias.
This study satisfied those criteria. It was a prospective randomized control study aiming for a scientific outcome, as opposed to the previous study, a retrospective meta-review focused on arithmetic. Along with invalidating the methodology generally, the researchers specifically invalidated 78% of the studies that comprised the
Invalidating this particular meta-analysis—cited 775 times in the academic literature alone—undermines the entire industry. I, myself, have invalidated it a different way, noting that the studies that underlie it (the majority of which used data from the 1990s and were authored by executives with strong ties to the wellness industry)
This NBER case was just the opposite.
Where does this leave the wellness industry?
It is safe to say that any debate on wellness savings is over: conventional wellness loses money in the commercially insured working-age population (also, according to a February Medicare announcement,
And just as the unique reporting relationship between the 2 opposing principal investigators lends great credibility to the most recent finding, another unique aspect of this ongoing debate lends great credibility to the conclusion of this article: I am offering a
_________________________________________________________________________
ii The other is measuring the decline in risk of high-risk members while overlooking any increase in risk of low-risk members. This is pure regression to the mean. Wellsteps CEO Steve Aldana was
iii See Figure 8, p. 46, with p. 77 in the online-only version making specific reference to the Baicker meta-analysis
iv As an example of such a lapse, the study also failed to disclose that 1 of the investigators had strong ties to the Obama administration, which, at the time the study was published, was attempting to build bipartisan support for the pending Affordable Care Act legislation. Allowing a 30% withhold for wellness was key to procuring the support of the Business Roundtable. (
Newsletter
Stay ahead of policy, cost, and value—subscribe to AJMC for expert insights at the intersection of clinical care and health economics.