We’ve had some recent posts (here and here) about the research of Brian Wansink, a Cornell University business professor who’s found fame and fortune from doing empirical research on eating behaviors. It’s come out that four of his recent papers—all of them derived from a single experiment which Wansink himself described as a “failed study which had null results”—were hopelessly flawed. I don’t know anything about the quality of Wansink’s other published work, but given the low quality of four papers which he advertised on his own blog, I’m concerned.
Yesterday we discussed the problems that can arise when doing quantitative empirical work with noisy data in the absence of substantive theory. I argued that the statistics profession is partly to blame for the attitude that many researchers display when they think they can make routine discoveries while at the same time having no theory of what is going on.
Today I want to discuss a slightly different topic, an issue that comes up implicitly whenever we criticize published work.
On statistical or methodological grounds, of course it is appropriate to point out flaws: this is how we as individuals and as a community learn to do better science.
But what about the good of society? Just as some people criticized us for criticizing the work of Ted-talk star and retired Harvard professor Amy Cuddy, on the grounds that Cuddy’s speeches have inspired millions even if her science is not completely sound, similarly one might criticize our criticism of Wansink, on the grounds that his books send positive messages about healthy eating behavior, even if his science is not completely sound.
Is Brian Wansink’s research doing the world more harm than good?
I have no idea. I assuming that Wansink genuinely believes that he’s making discoveries and that he’s helping people in his books, videos, interviews, etc.—and the research papers are part of that, for two reasons. First, publication in scientific journals is taken as a badge of quality. Without the peer-reviewed articles, maybe Wansink wouldn’t be on CBS news etc., he certainly wouldn’t be getting government grants, and his work wouldn’t be used to make policy. Second, he and others can use the specific claims in his research to make decision and policy recommendations.
Here are a few scenarios by which Wansink could be doing more good than harm, in spite of all the problems with his research methods:
1. One possibility is that some of Wansink’s work is of high quality. Sure, those four papers are pure noise, and I wouldn’t trust a thing in them—really it’s no better than flipping a coin a few hundred times and spinning stories about the patterns you see—but it could be that he has lots of other papers that are good, and maybe the low-quality recent work is just him trying to keep up his research productivity after having run out of good ideas. Kinda like how some authors will keep coming out with a book every year or two, even after they’ve run out of anything to say. If this is the case, it could be that Wansink is still doing more good than harm, if his influence on policy is based on his earlier, high quality work.
2. Another possibility is that the work is all noise, but that Wansink and his colleagues have put together noisy, meaningless research results in a way that makes a coherent and scientifically true story. I think this really could be happening. The idea is that Wansink and others, based on a mix of insight and decades of careful qualitative observation and experiment, have come up with a good understanding of why we eat the way we do, and then they use these experiments as a way of filling in the picture. From the point of view, the point of something like the quantitative analysis of pizza-restaurant experiment was not to learn anything new, but rather to come up with further illustrations of an existing storyline, and also to stimulate new insights. This is similar to the idea that an astrologer or fortune-teller might actually be able to give good advice, with the tarot cards or tea leaves serving as a pretext or even a stimulus for real insights.
3. A third possibility, slightly different possibility is that the quantitative and the qualitative research is entirely irrelevant, but it’s still all good because the existence of the publications and the big-money consulting is being used to bolster common-sense messages regarding mindful eating and small portion sizes. Wansink’s research could be doing more good than harm in that, by strengthening his reputation and that of his lab, the research-cred enables him to go on TV and sell books encouraging people to eat sensibly.
And of course there are scenarios in which his research does more harm than good:
4. Researcher finds random patterns in noise, uses p-hacking and salesmanship to get published; published work is believed and is translated into policy; people follow bad nutrition advice and their health is harmed or, at the very least, their quality of life is harmed because they feel they should be following some arbitrary rules. Wansink’s recent publications were in obscure places such as the Journal of Sensory Studies, but unfortunately we can’t really ignore them because Wansink apparently really is an influential person in the area of food research, so his noise mining could end up becoming real policy.
5. Even if the effect of the research claims is neutral—thus, not advice that hurts but advice that has no consistent effect—the entire Cornell Food & Brand Lab enterprise could do more harm than good if it takes resources from more worthy endeavors. These resources include government grants, journal space (representing the attention of the scientific community), public attention, and corporate funding. Related to this is the idea that bad research poisons the well of trust in science, making it harder for more careful research to get respect and attention.
I’m not saying these five scenarios are equally likely. Indeed, to say so would miss the point, as I think all of them are true to some extent. It’s hard for me to sum them up because I don’t have a good sense of their relative importance.
Does our criticism of Wansink do more harm than good?
Again, I don’t know. I’ll try to approach the question by considering the five scenarios above.
1. Suppose Wansink’s earlier work was of high quality and that his policy influence comes from that work. In that case, our criticism could be a bad thing, in that by discrediting Wansink, it could reduce the impact of that early work.
2. Suppose Wansink has been, consciously or unconsciously, using his qualitative understanding to build a true picture of the world. In that case our criticism should help him, and should help his research field, by making him aware that this is his method, and pushing him toward more effective use of his qualitative data.
3. Suppose the research is irrelevant but has been used in a good way to bolster solid, common-sense advice about eating. In that case our criticism could be bad in the short term (in discrediting this particular carrier of the healthy-eating gospel) but I hope it would be positive in the longer term, by motivating people in this field to do some serious quantitative research on the topic.
4. Suppose this low-quality published research is actually doing harm, that people are making decisions based on exaggerated claims coming out of noise. In that case our criticism should be helpful in warning people away from the bad work.
5. Suppose the main effect of Wansink’s work is to suck the oxygen away from serious research into eating behaviors. Then I’d hope our criticism does good (by discouraging journals from publishing this work and discouraging news organizations from promoting it), but I could see how it could do harm, by discrediting scientific investigation more generally and thus dissuading policymakers from making use even of high-quality empirical work.
P.S. The concern is not just theoretical. For example, I see that Brian Wansink is delivering the Dr. Robert C. and Veronica Atkins Foundation Curriculum in Metabolic Disease Lecture next week at Cornell Medical College next week. Perhaps he’ll tell the doctors there about ways in which his pizza restaurant research is relevant to metabolic disease. Since this is such a select audience, maybe he’ll reveal to them his Plan A that he’s been keeping secret all this time!