Currently Viewing:

NBER's Invalidation of Wellness: Behind the Headlines

Al Lewis wears multiple hats, both professionally and also to cover his bald spot. As founder of Quizzify, he has married his extensive background in trivia with his 30 years experience in healthcare to create an engaging, educational, fully guaranteed and validated, question-and-answer game to teach employees how to spend their money and your money wisely. As an author, his critically acclaimed category-bestselling Why Nobody Believes the Numbers, exposing the innumeracy of the wellness field, was named healthcare book of the year in Forbes. As a consultant, he is widely acclaimed for his expertise in population health outcomes, and is credited by search engines with inventing disease management. As a validator of outcomes, he consults to the Validation Institute, part of an Intel-GE joint venture.
The behind-the-headlines tour of the NBER study yields the most important conclusion
The focus on the headlines has obscured the most important conclusion from this study, which is that the entire participant vs nonparticipant methodology is invalid. Until now, that methodology had been 1 of the 2 methodologies the industry uses to reliably “show savings” in virtually any circumstance.ii

Of the 2, the participant vs nonparticipant methodology is the only one that arguably needs scientific proof along with mathematical invalidation. The mathematical invalidation has already been addressed on The American Journal of Managed Care blog. That study was a meta-review of 3 participant vs nonparticipant studies, which had a known benchmark—including the 1 highlighted above where the “benchmark” is that a wellness program can’t show savings before it exists. Therefore, the known benchmark is 0% savings on participants, invalidating the participants vs nonparticipants result showing almost 20% savings.

The best confirmation of any initial conclusion is an analysis reaching a similar or identical conclusion—but conducted using a completely different approach, by completely different researchers, with no relationship to the original researchers, and no confirmation or investigation bias.

This study satisfied those criteria. It was a prospective randomized control study aiming for a scientific outcome, as opposed to the previous study, a retrospective meta-review focused on arithmetic. Along with invalidating the methodology generally, the researchers specifically invalidated 78% of the studies that comprised the “Harvard study” meta-analysis, whose very widely publicized 3.27-to-1 ROI turbocharged growth of this field.iii

Invalidating this particular meta-analysis—cited 775 times in the academic literature alone—undermines the entire industry. I, myself, have invalidated it a different way, noting that the studies that underlie it (the majority of which used data from the 1990s and were authored by executives with strong ties to the wellness industry) were themselves riddled with obvious fallacies (including the fallacy that the aforementioned study can’t attribute a reduction in the cost of diseases of blood-forming organs to an intervention that had nothing to do with blood-forming organs). Yet, despite the obviousness of the fallacies, I could be accused of “investigator bias,” due to my history of exposing ethical and economic lapses in this field.iv

This NBER case was just the opposite. The principal investigator, Damon Jones, PhD, is an associate professor at the University of Chicago Harris School of Public Policy, where the author of the Harvard study, Katherine Baicker, PhD, is now a dean. This juxtaposition confers a rarity of rarities—reverse “investigator bias.” A researcher must be quite confident of his findings in order to publicly invalidate his supervisor’s most often-cited paper.

Where does this leave the wellness industry?
It is safe to say that any debate on wellness savings is over: conventional wellness loses money in the commercially insured working-age population (also, according to a February Medicare announcement, in the senior population, as well).

And just as the unique reporting relationship between the 2 opposing principal investigators lends great credibility to the most recent finding, another unique aspect of this ongoing debate lends great credibility to the conclusion of this article: I am offering a 7-figure reward to anyone who can show wellness doesn’t lose money. Despite loosening the rules in 2017 so that the burden of proof shifts to me, and agreeing that I can appoint only 1 of the 5 judges, no one has attempted to claim it.   

ii The other is measuring the decline in risk of high-risk members while overlooking any increase in risk of low-risk members. This is pure regression to the mean. Wellsteps CEO Steve Aldana was compelled to admit as much when he was caught doing it. As a hypothetical illustration of this fallacy, suppose the only risk factor is smoking. Further assume every employee smokes for a year but also quit for a year, and then repeats the cycle. Using the Wellsteps methodology of measuring only high-risk employees would show a 100% decline in smoking every year, while the actual smoking rate in the company remains unchanged at 50%.

iii See Figure 8, p. 46, with p. 77 in the online-only version making specific reference to the Baicker meta-analysis

iv As an example of such a lapse, the study also failed to disclose that 1 of the investigators had strong ties to the Obama administration, which, at the time the study was published, was attempting to build bipartisan support for the pending Affordable Care Act legislation. Allowing a 30% withhold for wellness was key to procuring the support of the Business Roundtable. (This linkage did not become widely known until 2013.)

Copyright AJMC 2006-2018 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up