Science is, in many ways, the best method we’ve found to identify truth. Humans are terrible at getting to the bottom of things on their own—we’re biased, and can be easily tricked by our own expectations. As I like to say, it’s easy to be confidently wrong if you only ever consider evidence that supports your prior beliefs. The entire idea of scientific experimentation is to remove that humanity from the equation and, in the grand tradition of Descartes, stick only to what we can empirically prove.
The problem, of course, is that science is still done by people, and people are flawed.
One massive issue in science—and academia more broadly—is the idea of correction. The grand old myth is that science is self-correcting. Horrifically flawed papers don’t make it past peer-review, or they are trashed immediately on release to the scientific community.
The reality is far more depressing. Scientific correction is a painful, years-long, opaque process that barely works at the best of times. I’ve got two recent cases from my weekend job as an expert in scientific error detection that I thought I’d share with you to show just how painful this whole process can be. Both of these papers were hugely impactful in their own way—one is now retracted, the other might as well have been.
The first case is this 2003 randomized trial on cinnamon for diabetes. With nearly 2,000 citations according to Google Scholar, it’s by far the most academically important paper that I’ve ever worked on. This is a measure of how many other papers have referenced this trial, and having 1,700 citations puts it in the top 1% or so of all papers ever published.
This sort of metric sounds dry and boring. Academia in general is a tedious world filled with red tape and exhaustion. But in medicine, those citations have IMPACT. There are clinical guidelines which real doctors use to treat their patients which talk about this amazing trial.
At face value, the paper is quite remarkable. The authors randomized 60 people with diabetes in Pakistan to receive one of three placebos or one of three doses of cinnamon extract. The results were astonishing - every group who received cinnamon had massive improvements in their blood sugar and cholesterol. The benefits were substantially bigger than those seen in modern drugs like Ozempic and Mounjaro. If true, this would be one of the most amazing findings in modern science.
But, in 2023, I noticed some extremely weird features of the paper. There are a few deeply problematic things, but one in particular is quite easy to spot. Have a look at the below table and see if you can tell what’s wrong here, remembering that this is meant to be from a randomized trial:
The problem here is that the groups have enormous differences at baseline. True randomization can result in some divergence - in fact, it is rare that there are no differences at all and papers with groups that are too similar are also suspicious - but in this case the chances that you would see this spread of glucose levels before cinnamon intake is literally billions to one. Just the difference between groups 1 and 6 alone has a probability of appearing once in every 2 million randomized trials or so.
So I emailed the journal with this and some other concerns. And waited. And waited. I followed up three times, most recently in early 2025 when I was told that there had been some delays but the journal was now going to take action. Finally, on August 20th 2025, the journal published this expression of concern about the paper which states that my concerns were verified by independent statisticians. I think it’s worth reading in full, but to briefly summarize:
The data in the study is completely shot.
The journal couldn’t get in contact with the authors.
The study is probably borked, but it’s impossible to know.
In cases like this, we simply do not know exactly what happened. Everything from poor data management to fraud could in theory explain these results. That being said, the study is completely worthless as evidence despite being cited by thousands of scientists and doctors for over 20 years. There are people who today recommend cinnamon instead of Ozempic based in large part on this paper.
This brings us to the second case of academic inertia. In April 2022, a recently-published study went viral for showing a remarkably strong association between vitamin D and COVID-19 outcomes. This was before we had strong evidence showing that vitamin D supplementation was not useful for COVID-19 infection, and it created a lot of buzz. At the time I wrote about the paper on Twitter, it had already been viewed 1 million times.
Most academic papers get fewer than a thousand views. 1 million is…very rare, even during the pandemic.
The paper itself was actually not that interesting. A team from Israel had taken a small sample of 253 patients who they had vitamin D levels for from before the pandemic, and looked to see if their vitamin D levels were associated with their risk of dying from COVID-19. Even in a best-case scenario, it’s a small sample and not really a useful study.
But there were other, more serious problems. The authors had uploaded their dataset online, and it was deeply bizarre. For example, pretty much everyone who had chronic obstructive pulmonary disease also had high blood pressure. It’s common to see overlap there, but almost unheard of to get 100% overlap.
In addition, the effect of having “low” vitamin D levels was mostly based on people who had exactly 10ng/ml. People who had 9.9 or 10.1ng/ml actually had a significantly lower risk of dying than the 15 people who had exactly 10. This isn’t evidence of malfeasance, but it’s hard to explain if the data didn’t have any errors in it. At the time, another commentator on Twitter said that the data was “made up”.
The journal that the study was published in, called PLOS One, replied on Twitter to say that they were going to investigate the matter internally. They then emailed me in May 2022 to say that the investigation was ongoing. I emailed them back twice, most recently in 2023, asking if the investigation was progressing and what form it was going to take. I never heard back.
On the 8th of September 2025, with no fanfare or further communication, PLOS One retracted the paper. The reasons given for this retraction are:
The inclusion of only patients for whom a vitamin D determination had been performed before COVID-19 infection introduces bias.
The potential medical reasons for a vitamin D evaluation prior to infection and the potential corrective therapy that may have been undertaken introduce major confounding factors that have not been adequately accounted for. This bias is further exacerbated because the vitamin D measurements used in the analyses were collected between 14–730 days before COVID-19 infection, without consideration of the potential changes in vitamin D levels — other than seasonality — in between the time of determination and the time of infection.
They then emailed me a few days later, apologizing for not communicating in over 2 years, and letting me know that the study had been retracted.
Now, if you look at the reasons above and compare them to my Twitter thread, you’ll notice that they are very different. The reasons that PLOS One has given for retracting the paper are very basic things that anyone who had read the manuscript would immediately know. The inclusion criteria and timing between vitamin D test and COVID-19 infection were stated plainly in the methodology section. These issues certainly do undermine the paper, but they’re also blindingly obvious things that the journal clearly should’ve noticed before publishing the thing.
Of the data irregularities, there is no further comment. It took three years to realize that there were basic biases in the paper and decide to retract. We don’t know if PLOS ever looked into the data issues. I emailed them to ask, and was told that “After PLOS became aware of the concerns, the article was reassessed by an independent member of the PLOS One editorial board and an independent statistical expert.”. What this means is, of course, anyone’s guess.
These stories are, unfortunately, the norm for academic error-correction. Both studies were, in different ways, irretrievably broken. In both cases, it’s impossible to know exactly why, but the data was sufficiently problematic that no one should have ever used them as evidence for anything.
The issues in these studies were also glaringly obvious. Anyone who had spent more than a moment of thought would’ve noticed the massive differences in the cinnamon study, and the vitamin D study stated the biases that caused its eventual retraction right there in the methods.
And yet, it took years to get any action on either paper. This is an all-too-common story in academia. Journals that will happily review and publish an article within weeks will take years to address even the most catastrophic errors in research that has already come out. I have many more cases of such issues that have simply never been acted on at all. As james.claims says, they are in the publishing business not the unpublishing business.
Both of these studies were impactful. There are people who take cinnamon for their diabetes today who are relying on bad data from the Pakistani study. There are people who took vitamin D for COVID-19 because of the Israeli one. Peer review is an inherently flawed system, which means that some bad studies will inevitably come out. But we could do so much better at dealing with errors and other issues than we currently do.
Something needs to change.