Over the past decade, it became apparent that a number of fields of research had some issues with replication. Published results didn’t always survive attempts at repeating experiments. The extent of the problem was a matter of debate, so a number of reproducibility projects formed to provide hard numbers. And the results were not great, with most finding that only about half of published studies could be repeated.
These reproducibility projects should have served a couple of purposes. They emphasize the importance of ensuring that results replicate to scientific funders and publishers, who are reluctant to support what could be considered repetitive research. They should encourage researchers to incorporate internal replications into their research plans. And, finally, they should be a caution against relying on research that has already been shown to have issues with replication.
While there’s some progress on the first two purposes, the last aspect is apparently still problematic, according to two researchers at the University of California, San Diego.
Word does not get out
The researchers behind the new work, Marta Serra-Garcia and Uri Gneezy, started with three large replication projects: one focused on economics, one on psychology, and one on the general sciences. Each project took a collection of published results in the field and attempted to replicate a key experiment from it. And, somewhere around half the time, the replication attempts failed.
That’s not to say that the original publications were wrong or useless. Most publications are built from a collection of experiments rather than a single one, so it’s possible that there’s still valid and useful data in each paper. But, even in that case, the original work should be approached with heightened skepticism; if anyone cites the original work in their own papers, its failure to replicate should probably be mentioned.
Serra-Garcia and Gneezy decided they wanted to find out: are the papers that contain experiments that failed replication still being cited, and if so, is that failure being mentioned?
Answering these questions involved a massive literature search, with the authors hunting down papers that cited the papers that were used in the replication studies and looking at whether those with problems were noted as such. The short answer is that the news is not good. The longer answer is that almost nothing about this research looks good.