Two decades of Alzheimer’s research was based on deliberate fraud
wallstreetpro.comTwo days ago: https://news.ycombinator.com/item?id=32183302
A month ago: https://news.ycombinator.com/item?id=31828509
Thanks! Macroexpanded:
Potential fabrication in research threatens the amyloid theory of Alzheimer’s - https://news.ycombinator.com/item?id=32183302 - July 2022 (236 comments)
Alzheimer’s amyloid hypothesis ‘cabal’ thwarted progress toward a cure (2019) - https://news.ycombinator.com/item?id=31828509 - June 2022 (307 comments)
How an Alzheimer’s ‘cabal’ thwarted progress toward a cure - https://news.ycombinator.com/item?id=21911225 - Dec 2019 (382 comments)
The amyloid hypothesis on trial - https://news.ycombinator.com/item?id=17618027 - July 2018 (43 comments)
Is the Alzheimer's “Amyloid Hypothesis” Wrong? (2017) - https://news.ycombinator.com/item?id=17444214 - July 2018 (109 comments)
I recommend reading the comments on AlzForum [1]. From the discussions, it sounds like this fraud is significant in terms of Dr. Sylvain Lesné's work, but that the news has been vastly blown out of proportion.
These comments are written by real Alzheimer's researchers. They all disagree with the notion that Lesné's papers have been important to the field, and therefore undermine the idea that this has any bearing on "two decades of Alzheimer's research". (Karen Ashe, co-author of the main paper referenced here, also stops by the thread.)
[1] https://www.alzforum.org/news/community-news/sylvain-lesne-w...
I agree, the discussion here is reassuring. This may just be a case of a fraudulent paper piggybacking on an otherwise credible research program focusing on amyloid oligomers - not the plaques which recent failed therapies have targeted, and which many of these researchers knew were poor targets for decades before the FDA debacle over aducanumab.
We can blame regulatory capture of the FDA for approval of failed drugs, rather than the scientific establishment. And we can blame Sylvain Lesné for Sylvain Lesné's fraud.
I would encourage those of us whose only knowledge of this topic is the word "amyloid" (I admit I am one of them) to read the scientists' comments and appreciate that there is more to this than we know. There are complexities, nuances, diverse perspectives and healthy disagreements. It's not just a political battlefield. Projecting culture war into it would be harmful to the scientific progress we all value and to the millions who suffer from Alzheimer's.
Their comments are not without self-interest. They all see the potential of their own funding being re-evaluated if they admit that any of their ongoing research build's on (or even references) Sylvain Lesné's work.
They have an interest in limiting the scope of the perception of fraud in the field.
And the people who want a change in direction have an interest in calling out any of that sort of thing that may be happening. So if it is, I hope to see it called out in a scientifically substantial way.
I recommend reading these instead:
Alzheimer’s amyloid hypothesis ‘cabal’ thwarted progress toward a cure (2019) https://news.ycombinator.com/item?id=31828509
How an Alzheimer’s ‘cabal’ thwarted progress toward a cure https://news.ycombinator.com/item?id=21911225
The amyloid hypothesis on trial https://news.ycombinator.com/item?id=17618027
Is the Alzheimer's “Amyloid Hypothesis” Wrong? (2017) https://news.ycombinator.com/item?id=17444214
Why, at this point should we believe any one scientist writing in that forum wasn't already sunk far deep into amyloid research in their career?
Not to mention that after a quick glance on the comments section, I fail to see where you get the idea that "They all disagree with the notion that Lesné's papers have been important to the field". Apart from the very first comment from karen Ashe (who will obviously be defending her research) and a few other who's working on related topics, other commenters seems to be keeping their suspicion at amyloid hypothesis.
Rather than just scrutinizing a single hypothesis for AD we must understand that it is a multi factorial disease and that there are other hypotheses as well (i.e., acetylcholine, tau, transport-clearance), which affect and complicate this disease. Furthermore, we know very little about the brain, hence why novel therapies consistently fail in the clinic compared to other diseases (i.e., cancer). Most of these antibodies that target amyloid don’t even cross the blood-brain barrier (<0.1% and need an effective dose of 1000x blood-plasma for effective amyloid lowering) - provided with this, can we really attest the failure of the amyloid hypothesis? Moreover, the commenters there are big names in the AD community
And how many of those commenters have a vested interest in the current research direction?
It's pretty disrespectful to signal (without evidence or elaboration) that researchers are not credible (or worse, broadly lying) in order to keep their research grants flowing. A hypothesis that turns out to be wrong is something both industry and government are obviously not going to invest further in. The people working in the field have skills to transfer to other departments and projects; these aren't the sort of scientists and engineers that are out of ideas or work to do.
It's also plenty obvious that there is no single, monolithic "current research direction" or even that this researcher's work was of fundamental impact when it was published - not to mention the number of people that were highly skeptical from the beginning.
Based on the way that string theory keeps on going like a zombie in physics, I don't think its disrespectful at all to consider allegations that are pretty much indistinguishable from those. Although I think that heavily entrenched groupthink and biases are all that is required. I don't think they're consciously think that "I need to lie in order to keep the research grants flowing" but its instead that old adage that "It is difficult to get a man to understand something, when his salary depends on his not understanding it". To an external observer those are largely indistinguishable and have the same effect, but in the latter case the person has convinced themselves of the lie or the half-truth via financially inclined self-deception.
I think there is truth to the general principle you refer to, but I don't think it accurately describes what I saw skimming experts' comments in the linked thread. I'm an outsider to medical research but have experience in other parts of STEM research at universities. Here I saw a plenty of nuance, documentation of historical skepticism, concern over broad perception, and plenty disagreement over technical points. Far from a unified kool-aid drinker sort of situation. And I think there has been plenty of changes of opinions in the Alzheimer's field in recent years given the number of failed drugs - which goes against the idea that these scientists are following their career over the evidence.
But people who are motivated by a decade or two of having their salaries paid by the leading hypothesis aren't going to produce unified kool-aid drinker kinds of rationales to support it. They've had decades to internalize the arguments and they will be nuanced and multifaceted. They're experts and their defense of the old paradigm will look just like expert opinion. There can still be rotten core foundations at the bottom of it all.
I'm not claiming there are no overly stubborn PIs. But we should not be lazy and paint so a brush that we view the field as a monolith.
Nobody is actually arguing that everyone in the field is goosestepping in perfect synchrony like a cartoon.
But human bias applied by groups is actually very powerful.
It's not disrespectful to point out conflicts of interest. Recusing yourself is standard practice for good reason.
Harboring skepticism of the work people did with a seemingly fraudulent researcher is a good idea. Dismissing everybody in a field whether or not their work is fraudulent is disrespectful approach (not to mention useless).
It’s not even a good faith argument, people cannot prove they’re not biased.
It’s just your basic lazy conspiracy theory thinking dressed up as cynical aloofness.
Given everything we know about how research funding has been captured and shaky ethics in particular domains, skepticism is warranted, not disrespectful.
I'm not asking for people to turn off skepticism or blindly trust researchers. It's not disrespectful to be skeptical.
What is disrespectful is not bothering to read what people have to say before dismissing them as liars who are too vested in "the current research direction" and/or money for their perspectives to matter. It only takes reading a few comments to see that's not happening - for starters, people were skeptical of this group's work for a while now.
Also, “believe science” is the exact opposite of the scientific method.
“Hurl unfounded and baseless accusations of personal corruption” is not the scientific method either.
Trying to pass off conspiracy theorist “disprove the negative” arguments as reasonable discourse is intellectually lazy, at best.
Is that an accusation of bad faith or just a non sequitur?
There is that minor issue that we don't have nearly the same experience and resources as the scientists who say what they say. Are you implying we all becom biologists and figure out the source of Alzheimer's ourselves? As some point you have to trust people and choose someone.
lol
See related :
"Alzheimer’s amyloid hypothesis ‘cabal’ thwarted progress toward a cure (2019)"
https://news.ycombinator.com/item?id=31828509
https://news.ycombinator.com/item?id=21911225
(also see dang's comment who lists 2 more thread https://news.ycombinator.com/item?id=32213973)
Parent comment is downvoted but it seems like most alzheimer's researchers have vested interest in amyloid hypothesis one way or the other.
Can we replace the link with the original Daily Kos article [1] that this is a copy-paste of? I have no idea what wallstreetpro.com is, but it looks like blog spam.
[1] https://www.dailykos.com/stories/2022/7/22/2111914/-Two-deca...
Actually, I believe the original would be the Science article [1]
[1] https://www.science.org/content/article/potential-fabricatio...
Depends on what you mean by original. The Daily Kos is reporting on the same news, but they are not the same articles.
-- the article in science a couple of days ago is a bit more comprehensive however the submission is easier for quick digesting --
https://www.science.org/content/article/potential-fabricatio...
This isn’t supposed to be possible. This is soo much public money wasted.
-- from reading - the problem was there are no imaging experts - so basically all images in scientific publications are not scrutinized for photoshop - good thing this guy didn't have access to dall-e --
That's not really the problem. There are many ways to fake papers and there's no guarantee that a Photoshopped image is detectable just by looking at it.
The problem is that somehow researchers managed to get hundreds of millions of dollars, entire labs, and 15+ years of research on the back of a study that presumably nobody ever tried to replicate, nor did they notice that the claims weren't true even when building new research on top of it. That's the problem and it is frankly very hard to understand how it's possible.
> basically all images in scientific publications are not scrutinized for photoshop
This was (mostly) true back then, but it is definitely not true today. People tempted to commit fraud now have to be worried about people like Elisabeth Bik exposing them and ruining their careers. In my experience, the type of people known lie in papers overlap strongly with those that are career-minded/money-driven. So having a few journalists with the skills to detect fraud is an obvious win. Some of the frauds will just get better at, that's just how it does, but it's not like there are no imaging experts in the field.
So the frauds of today are done in smarter ways.
Maybe they will be discovered with new tools developed in a few decades.
Hopefully so, but there will probably always be a back-and-forth between frauds and journalists in the same way security is always a competition. At least the easy frauds are more likely to be caught today.
The bit at the end about the NIH continuing to allocate funds to the exposed researchers... with the program director being one of the authors of the exposed paper - that stings. But it also shows how deep corruption and "networking" runs in places like this.
This is a tragedy.
Academia and research needs a new broom. Presently incentives are peverse. Impact factors, publisher corruption, grant applications and funding are a blight on science.
Kickstarter for science ?
I actually freelanced on a project 10 years ago which was exactly that. Don’t take its failure as proof that such an idea cannot succeed, but the major headwinds were that most research is pretty difficult to explain (and scientists are not selected for their ability to communicate with Lee people).
Also remember that Kickstarter found it necessary to write a blog post “Kickstarter is not a store“, because people were expecting projects to return fast, reliable, tangible results. By its nature, science will be even further from that revealed preference of funders
Maybe that is the business case ? Making science accessable and in return get ppl that care or help with funding ?
Neither solutions are perfect as most things in life, I guess the question is, will it be better enough compared to what we have now ?
Minor spelling point, it's "lay people" not "lee people".
... because there are never fraudulent Kickstarters?
You have a better idea to contribute ?
This is happening across every single industry though.
My opinion: capitalism has corrupted literally every single thing it has touched.
Yeah but that's true for basically every politico-economic system. Should we go into lysenkoism or mao telling people to kill sparrows, or Sri Lanka stupidly banning non-organic farming, or the EU ending nuclear and adopting coal?
You got the first two right, and the second two completely wrong.
Well, keep in mind that in this particular case, it was capitalist attitudes that exposed the fraud. From the dailykos article:
> "The suspicion that something was more than a little wrong with the model that is getting almost all Alzheimer’s research funding ($1.6 billion in the last year alone) began with a fight over the drug Simufilam. The drug was being pushed into trials by its manufacturer, Cassava Sciences, but a group of scientists who reviewed the drug maker’s claims about Simufilam believed that it was exaggerating the potential. So they did what any reasonable person would do: They purchased short sell positions in Cassava Sciences stock, filed a letter with the FDA calling for a review before allowing the drug to go to trial, and hired an investigator to provide some support for this position."
However, the desire to gain a profit by pushing a questionable drug through trials was also involved. I'd note however, that in the Soviet Union, the likes of Lysenko also pushed fraudulent research in order to improve their standing in the Soviet heirarchy, which came with various rewards.
People like wealth and power, and some will do anything to get it, regardless of the nature of the society they live in.
This isn't capitalism; it's nepotism and cronyism, and those are not the same as capitalism. Capitalism is just a means of generic exchange, it's a hammer that can be used to build something or bash in someone's skull.
Forgive me, I don't think the corruption necessarily follows from "capitalism". Corruption is a thing that stands alone, and it is multi-faced: Some is greed. Some cowardice. Callousness and lack of compassion. Self hate, lack of good character and backbone. Then there's laziness and sloth. We call it "convenience" today. And a measure of self-importance, egotism, or pride. We can't point the finger at one political ideology that's only a few hundred years old. Start with Aristotle, the cardinal virtues, or with Dante or Cadmus on "sins". Read any religious text. Or as commenter here reminded me today, dig into anthropology, which is fascinating.
What does this have to do capitalism?
Funding for these projects is often disconnected from the market.
They are usually dispersed by massive bureaucratic agents with incentive structures that have nothing to do with profit (more internal politics and prestige).
The perverse incentives still exist outside of capitalism. People, including scientists, do not like to admit when they are wrong. There is no incentive to retract a study. Bribery and corruption are not a trait of any particular economic system. It is easy to make criticisms but sometimes the cure is worse than the disease.
I feel like capitalism is a natural phenomenon of specialization.
Imagine if the human body didn’t have a dozen different organs each producing a specialized product for your body and instead had one organ that tried to do everything?
Black Plague was natural too. That’s no defense of it.
I'm curious, do you have a counterpart to "capital" and "property" in your analogy?
I think capitalism facilitates specialization - and that quite effectively at scale. Specialization can occur without capitalism, too. There are other systems to coordinate specialists.
Of course -- just look at all of those successful centrally planned economies throughout history that definitely did not regularly subject their population to famine
Capitalism has produced more science and technology than any other system. It’s not a coincidence that the Industrial Revolution didn’t happen until capitalism was the prevalent Western system.
What you wrote is consistent with the comment you replied to
Just because capitalism has generated good things doesn’t mean it’s perfect. We need to acknowledge the damage it’s causing and think of ways to move beyond it.
Unless you want this to be the way things are for the rest of humanity.
I've learned that to some people "capitalism" is just shorthand for the dark force behind all evil, and functions a bit like "the devil" in Christian tradition.
Capitalism? Do you think it was all way better in Soviet Russia?
Blaming “capitalism” is just the “old man yelling at cloud” if you are 20.
You can decry Western Capitalism without lauding dictatorship with a communist twist.
Some people latch onto whatever they can to gain power to exploit people.
Using failed dictatorships to prove 'communism is worse than capitalism' (which in the West we heavily curtail, but only to the point where we avoid uprisings against those holding power) is exactly the sort of shallow, illogical thinking you seem to rail against.
Communism fails because it doesn't account for human greed; it assumes everyone is onboard with improving society. Capitalism succeeds inasmuch as it panders to human greed to the point of evil and ignores the vast majority whom it fails.
Mind you, anarcho-syndicalism is where it's at, you can expect to wield supreme executive power just because some water tart gives you a sword ... /montypython
Yes, economic systems are completely binary. Just like political ones. Either everybody votes on everything or absolute monarchy. Wait...
There's a cogent argument to be made that under capitalism, once market saturation is reached, any further excess wealth (and capitalism creates lots of that!) is redirected to zero-sum competition (which is, on a societal level, waste).
I don't think capitalism corrupts everything it touches - but I'm pretty sure it corrupts saturated markets, and these days that's an awful lot of things.
> There's a cogent argument to be made that under capitalism, once market saturation is reached, any further excess wealth (and capitalism creates lots of that!) is redirected to zero-sum competition (which is, on a societal level, waste).
I think that's a strong take and agree somewhat. It's not at odds with the fact that capitalism has created more innovation, wealth and raised standards of living more than any other period in history. The problem is not capitalism, it's that capitalism is over.
In the long tail of diminishing returns all that's really left for people holding obscene sums of money to do amidst shocking inequality is to lie, cheat, steal and rat-fuck one another over the remaining opportunities to die on the biggest pile of ostentatious wealth.
Capitalism was a great system that ran out of fuel. I don't think we expected it to stall so soon, but it's hit some internal limit. We should be as worried about that as climate. These problems are bound up together. If we are going to preserve the liberty, opportunity, and democracy that have naturally ridden along with it we had better figure out a way to creatively re-invent the industrious ethic underpinning capitalism because, frankly, the Chinese model of consumer-communism is nothing to celebrate or hope for.
I think it's fair to blame this one on capitalism-like structures without implying that the way to solve this is communism.
Clearly institutional competition for grant money has increased and changed what it means to be a successful researcher. Incentives are displacing virtues. But it's hard to find incentives, especially financial ones, that are fully aligned with honest research. Maybe we are trying too hard to find those.
Capitalism-like structures? The corruption here is fully within academia, which is a non-capitalist reputation economy running parallel to the real one. Absolutely nothing about this event has anything to do with capitalism except the fact that capitalist incentives and systems (short selling) led to its exposure.
Really, the whole point of academia is that it's not capitalism. It's much closer to communism. Resources are allocated by committee according to the priorities of the state. There are no price or market signals anywhere, and who gets ahead is largely related to how well they project their own work upwards. Plus of course, it seems to be strongly ideological. Academia is completely dominated by the left to the extent that in some departments there are almost no conservatives at all.
This is very similar to another scientific fraud case from two decades ago, involving Bell Labs and Jan Hendrik Schön, and a series of major papers published in Nature and Science from 1998-2002, and described in an excellent book, Plastic Fantastic: How the Biggest Fraud in Physics Shook the Scientific World by Eugenie Samuel Reich.
https://physicstoday.scitation.org/doi/full/10.1063/1.324848...
A key point, from the above review, that I think explains a lot of this behavior:
> "Reich points out that fraudsters like Schön could get credit for “first discovery” if, before they are caught, their false claims are confirmed by others on the basis of genuine data."
It did shake up the field of organic semiconductor device research in physics by increasing scrutiny and changing some requirements (for example, electron microscope imagery of claimed devices is now a requirement for publication). However, as the top post at present notes, the incentives are backwards in academic science these days, and the role of funding organizations and high-profile journals is as problematic as that of the originating fraudsters.
Maybe this instance of fraud will do the same for the biomedical field, by forcing researchers to release their raw data and full-resolution images as a condition of publication, although that would require a major shift in behavior in today's patent-driven startup-centered heavily-corporatized biomedical research world.
Personally, I'll note that during the years I worked in academia, of the three PIs I worked with, I discovered two engaging in fraudulent research to greater or lesser extent. The main differences between them and the one who wasn't were (1) lab notebook discipline and recording and storing data securely, (2) in-house replication was required, (3) no toleration for BS and shady behavior. The others broke all those rules. (Unfortunately I picked the wrong PI to work with, and ended up leaving academia in a fit of contemptuous disgust.)
A good rule of thumb: If some research claim hasn't been replicated, and if the data and methods aren't transparently available, then it's as likely to be fraudulent garbage as not, and it's not worthy of further examination.
There is an excellent three part video series about the Jan Hendrik Schön fraud on YouTube: https://youtu.be/nfDoml-Db64
If what is presented in this article is actually true that would be a systemic failure of epic proportions, right? Not just in this concrete case but don't these fundamental papers get reproduced before everybody takes them for granted and pours in enormous amounts of research grants believing in them? Is this article an exaggeration?
As a rule, papers do not get reproduced. Reproducing papers is a lot of boring work that beings no benefit to oneself.
I'd like to know whether the police have ever been brought in for a scenario like this. It may not seem as clear cut as regular financial fraud, but the end goal is to further one's career, which results in monetary gain. As such, this does seem to fall within most countries' definition of fraud. For example, from the UK:
Fraud by false representation
(1) A person is in breach of this section if he—
(a) dishonestly makes a false representation, and
(b) intends, by making the representation—
(i) to make a gain for himself or another
In this conversation, it’s important to distinguish science itself (which, along with facts, are already under attack in our politics) from failings of the research industry.
Most interesting to me, is how if you get the model wrong, thousands of scientists will use it anyway, billions will be spent, but no one challenges it! Amazing.
It really lays open how easy it is to mislead everyone. All these siloed scientists won't have a clue anything is wrong. This is how conspiracies would work... if there is advantage to someone somewhere and they have the means to alter the model in their favour, why wouldn't they?
The linked article is sensationalized an disingenuously mixes the author's opinion with straight reporting on the Science article[0] previously discussed here[1]. Science never suggests that anybody other than Lesné was involved in falsifying results, and even then it is careful to not accuse him of deliberate fraud. Neither article suggests who the second scientist referenced in the headline may be.
I suspect there was deliberate fraud, but this article doesn't provide any more evidence of that than previous articles.
> Since that 2006 publication, the presence or absence of this specific amyloid has often been treated as diagnostic of Alzheimer’s. Meaning that patients who did die from Alzheimer’s may have been misdiagnosed as having something else. Those whose dementia came from other causes may have falsely been dragged under the Alzheimer’s umbrella.
I think the author is confused about the controversy he is reporting on. Nobody is suggesting that there aren't elevated levels of Aβ in Alzheimer's brains. The controversy is only about the presence of Aβ56, and as far as I know Aβ56 was never used to diagnose Alzheimer's disease. It should also be noted that this is only relevant to postmortem diagnosis, so even if they were testing for Aβ*56 it wouldn't have affected the diagnosis of living patients.
At the bottom of the article is a note, "Article written by Mark Sumner via Daily Kos". This explains a lot. Daily Kos is a site that got its start with sensationalized political articles. Now they've apparently expanded to subjects where they can do more damage.
---
As I post the comment, the title of the linked article is: "Two decades of Alzheimer’s research was based on deliberate fraud by 2 scientists that has cost billions of dollars and millions of lives"
[0] https://www.science.org/content/article/potential-fabricatio... [1] https://news.ycombinator.com/item?id=32183302
The most likely motive for this kind of fraud at this level is that the researchers “know” their hypothesis is correct and are just fixing some “mistakes” in the data.
People do this all the time. In fact I’d say it’s the usual reaction when confronted with data contradicting one’s beliefs.
Scientists are supposed to learn to go past that but I wonder how many actually do, especially when there is both social and economic pressure to conform to a school of thought.
I'm not sure that's the most likely motive.
I think people want it to be a worse motive because they don’t want to admit how irrational and deliberately ignorant the vast majority of humans are most of the time.
Looking at reality in an unbiased way and trying to draw rational conclusions is incredibly rare and requires effort. Ego and social group pressure are the enemy.
Cackling evildoers are also very rare. Most evil is a product of how we normally behave and of normal social and economic incentives.
It takes a lot of life experience for someone with integrity to find out that other people are simple dopamine hunting monkeys who will flip flop to whatever social pressure is being applied at that present moment. We assume that others are like ourselves.
And so the very people who are supposed to call out and fight the irrationality are actually blind to it. You do get people calling out the irrationality, but mostly they are driven by social incentives rather than duty and so they turn their eyes away from realms that are deemed pure. They have no incentive to look inwards to the scientific institutions because that way they lose the social esteem they gain from association with the scientific establishment.
This should result in a very long jail sentence and complete banishment from society
By far the most effective preventer of dementia appears to be vaccinations, with 40% effect size. It doesn't seem to matter what against. Second might be antivirals against herpes.
These people need to be formally immediately banned from any NIH activity, and criminally charged. We have known for years that their crap work was useless.
Got a source for this big and fascinating claim?
Alzheimer's risk 10 times lower with herpes medication
If it really doesn't matter which vaccine, is this a case of stirring up the immune system to get rid of anything the immune system recognizes as different? There are articles showing ivermectin helps shrink melanomas too. The leading hypothesis was that the immune system once woken does its thing. It does concern me that once woken it might also take offense to my pancreas or thyroid or gluten or...
Nobody knows. Nobody is dispensed anything, out of the $billions diverted to amyloid cultists, to find out.
Wow. That's impressive.
Though the effect is from "antiherpetic medication", not vaccines.
As I said in the original remark (which see), "Second might be antivirals against herpes."
This Twitter thread highlights that this is a hyperbole/misinformation: https://twitter.com/samuel_marsh/status/1550883405105168386
„I sincerely doubt that the absence of this particular paper and AB*56 from historical scientific record would have significantly changed the last 20 years of AD drug development. That is because there is strong genetic and other evidence for the role of amyloid in disease.“
That thread appears to be defending what appears to be one of the deadest ends ever in medical research. How much failure do these people need? I'm genuinely curious.
Reading it carefully, not really. The author is equally against anti-amyloid drugs as a solution to Alzheimer's--arguing that amyloid is at best a starting gun to the disease, and when it goes off, fixing amyloid won't reverse the disease.
Want to see amyloid defenders? Read comments on Derek Lowe's blog, and there's quite a few commenters there who are arguing that a better amyloid drug that actually clears out the plaques is needed (apparently not realizing that's what Aduhelm, and it still didn't work).
Eh, we know diabetes is often reversible with careful diet and there's plenty of amyloid deposits in that disease.
Evidence is mounting that amyloid is the downstream result of whatever actually causes these diseases.
Welcome to science folks. Having worked a decade around a campus regularly sitting in on presentations (and being in research myself) I can say I believe 80% of science wrong AND not reproducible, 10% being an outright fraud (10% being legit).
Highly recommend this book to discuss why: https://www.amazon.com/Rigor-Mortis-Science-Worthless-Billio...
It all comes down to incentives.
(1) As a researcher you lose funding m if you don’t produce
(2) funds aren’t allocated to reproduce
(3) Researchers who publish will block research that disagrees with their work (as they’re also reviewers) (will lose future funding / have more competition)
(4) Researchers wont rescind their work if later findings warrant it (no incentive to)
(5) N numbers are way too low (higher N is more money)
> I believe 80% of science wrong AND not reproducible
Maybe that's the case in your area of research. In mine (math, physics) it definitely isn't. So I would be a bit more careful about the wording here. ("80% of science" – what science?)
In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset bias or accidents upon patient matching algorithms or confounders.
P < 0.05 is outdated at this point, for anything truly ground breaking p < 0.005 or < 0.0005 is probably a better choice, and even then I would ask "Where did you get your dataset from and did you combine (!!!) datasets from multiple orgs."
P less than 0.0005 isn't much better because if that's the criteria then you still have to accept that psychic phenomena have been demonstrated in the lab.
The methodology is deeply flawed, given that you haven't stopped accidental p-hacking (publication bias) and deliberate p-hacking (researcher data mining, e.g adding endpoints after looking at results), which compounds to create fake results with astronomically tiny p-values.
You can only trust single, extremely large, canonical RCTs which were announced in advance, and in which you are confident there is no survivorship bias in terms of the possibility that this RCT would have been cancelled had the results been thought to be negative halfway through.
Epidemiology (victim of researcher p-hacking and impossible to deal with confounders) or meta-analysis of RCTs or single small RCTs or RCTs that weren't announced in advance (victim of publication bias) should be taken with a grain of salt. If you accept conclusions drawn from these things at face value, be prepared to accept anything, because the methodology you've accepted is proven to be easily capable of demonstrating any fake phenomenon as true.
I think you hit it exactly right.
There are so many ways that are studies can be flawed either accidentally or purposefully. I truly assume most studies are not purposefully fraudulent - people, by and large, are honest... but I do believe there are enough problems with our current methods that most studies are not truly accurate.
Having a publish clinical trial regimen - created before your study starts - helps with a lot of this. It answers how you segment patients, match patients, handle patient dropout, and specifies what you are trying to compare for outcomes.
Is it the end-all be-all? obviously not. I don't know what the true answer is.
What is the source for your belief? Something unscientific perhaps?
The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past.
Its actually my job.
Any study in life sciences will have lots of discussion of claims and modes for the future. But at the core will be the data, which is really the only thing that any researcher actually should trust from a paper.
What are you designing new experiments for? Are you taking results in one cell like and condition and trying to transfer them to new cell lines and conditions?
Are you trying to reproduce the same data in the same cel lines/organisms?
There's such a huge wide array of what your could be taking about, but based on my experience in life sciences what you say does not seem the least bit realistic, in my first interpretation of what you are saying.
The problem with getting more and more specific about my experiences and expertise is you get closer and closer to doxxing yourself. Honestly, it may be time to retire this account again.
Also its worth noting that I am not claiming my experience is the end-all be-all. I am stating that, from my experience, I have incredible distrust for many studies that are published with 'amazing' results until peer reviewed preferably on disparate datasets.
My experience is based around outcome based studies of the effect of drugs/treatment/regimens in oncology and oncology adjacent fields. This includes drugs treated alongside traditional cancer regimens to assist with managing adverse events and toxicities.
Other fields may not have this reproducibility problem. Mine does. Even if the study design if perfect, and I can't imagine most are, the data itself can be questionable.
Consider - what dataset would you use to identify if patients taking keytruda had a higher incidence of high blood pressure?
You can use data from an EHR, licensed for deidentified studies, but EHR data is a burnt down trailer park of questionableness and its use in studies has been laughed at in many conferences.
You can use data from individual enrolled patients (for a clinical study) but then the cost is extremely high vs a non interventional observational study using other data methods. The value of the data is likely to be higher, but since it costs more to collect maybe you are only in a few regions that may have a higher prevalence or incidence of this anyway. Troublesome.
What about insurance data? You can get it cheaply, if you have high blood pressure GOOD doctors are likely to medicate you with a drug meant to treat it, and you can get it across the country. Seems good right? And it is, as you can generally draw an implication of high-blood-pressure->treatment-with-drug-x. So for a yes/no study it can help, but what if the base condition causes high blood pressure and we want to tell if the drug causes a HIGHER amount of high blood pressure than others. Insurance data by itself may not be enough to tell this data.
So what do you do? You are stuck with no great answers.... and this is assuming your study design is perfect.
So you buy multiple datasets in some third party health marketplace, and someone gets the great idea to combine the datasets to increase the n value. Well, too bad those datasets have a high overlap. So you have attributed a higher power to the study than is relevent.
I hope this explains more about the concerns I have. Though, I suppose, it may mean that this account is now dead. I will have to think further. Anyway, hope you have a great day.
Your job doesn't cause you to examine enough papers to create a conclusion for all research being done around the world.
I know you are trying to say something, but I am not quite following what it is you are trying to say. Being more blunt may help.
You may disagree with the experiences I have had as it may not be in alignment with your thoughts and ideals, and it still doesnt make you a bad person. It doesn't make me a bad person either.
What they're saying is that the level of corruption and false claims varies dramatically by field. Biology/medical/public health/social science seems to be incredibly corrupt but e.g. computer science doesn't have the same issues in the same way. Even then you can drill down to sub-fields, so ML has bigger problems within CS than computer graphics does for example.
I didn't say you were a bad person. Your personal experiences in a field as large as "scientific research" offer you no useful information to judge that entire field.
Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124
This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.
This is assuming your experiment was designed to test your hypothesis and that you model your null hypothesis properly. A p value means nothing anyway until you have both of these. Add to that the fact that many wrongly consider that a high p value disproves the original hypothesis.
I think you got it - there are many ways for studies to fail and its hard, at this time, to validate that a study succeeded except multiple studies confirming the original finding.
Even then, I would worry that the results may be caused by some confounder in the original dataset/design instead of something you can trust.
I'm also in physics and I agree with this. There's no chance that 80% is wrong.
One factor contributing to this. In natural sciences, you take other people's papers as truth and build on that. In theoretical physics on the other hand the _first step_ is you reproduce their results.
There's limited value in reproducing a theoretical result. That doesn't tell us if it's actually a valid model of the world. You can "reproduce" the results of string theory all day long, but nobody know if that's how things actually work. It's entirely possible that 100% of the papers in that sub-field are wrong even if the math checks out.
Why do you think that is? (Serious question). My understanding in life science research is that as a rule, papers do not get reproduced, as there is limited benefit to the researcher. Why is the same not true in physics? What is so seriously flawed about life science research that is not flawed about physics?
Isn't it obvious?
In theoretical physics, one of a researcher's main concern is to not to make wrong claims. Granted this risk aversion has its own downsides, but the upside is that he is extremely careful on what he takes to be true. So it's not true that not reproducing the research you rely on has limited benefit. It has the huge benefit that the researcher convinces himself he's not introducing wrong premises. Research is already difficult enough without making mistakes in this silly way.
It depends on the type of research.
If your research is mathematical or involves a lot of engineering, you are building on earlier results. In many cases, your own results will depend on the correctness of earlier results in a measurable way. You end up replicating others' research without even trying.
Empirical research is harder. You cite earlier results, but there is no clear connection between their correctness and your results. Especially if effect size is small. Earlier research has more effect on the framework you use to interpret your results than on the results.
Academic researchers rarely replicate others' results, because it's expensive and not particularly interesting. People typically come to the academia because they want to work on something they personally find interesting. If you want to have experienced scientists working on something administrators tell them to do, you better pay industry-level salaries.
My guess: In theoretical physics, you reproduce by spending an afternoon manipulating equations. Maybe a few days at most. But you reproduce in life science you need to get test subjects and treatments, etc. For example, how would you reproduce that a diet high in olive oil reduces the chance of heart attack?
It's usually a lot more than "a few days at most". In my field it was often between a week to a couple of months. But you got the right idea.
Math isn’t immune to error and more commonly ambiguity. That said mathematicians tend to have an admirable diligence and sky high intelligence, so yeah there’s a ton of quality work. With rigorous formality and computer checking it’s conceivable that math as a field can become virtually flawless.
As for physics, well it depends. Laboratory physics has produced the finest predictions in any science by several orders of magnitude. Quantum Electrodynamics is freakishly accurate. On the other hand it’s hard for me to see cosmology, to gently pick on an easy target, as more than extremely well researched and plausible science fiction. Then you have particle physics which has excellent laboratory equipment and produced fantastic results, but which has, in the opinion of at least one elite particle physicist personally known to me, perhaps painted itself into a corner. The Standard Model is good enough for government work, but nobody believes it’s the best possible theory.
Well, math is one of those things where (in most cases) anyone (...with the right knowledge in the field...) can verify the results (mostly) relatively easily and cheaply.
For physics, it depends.. is it just "applied math" (so, verified easily), is it a CERN-type (LHC,...) experiment (hard to replicate, unless well.. you work at CERN), where many many people process the data, or is it something that is done only in your "lab", and hard for others to replicate.
On the other hand, finding thousands of patients and running a study is practically always hard and expensive.
“Oh, people can come up with statistics to prove anything, Kent. Forty percent of all people know that.”
Wow, sounds like the Pareto Principle is alive and well in the arena of Science as well.
That’s the last 10% they left on the table. That last 10% makes up 100% of legit science.
Math is by its nature falsifiable. Much of what is considered “science” today is not. This is often the stuff that’s used as a political weapon via smug “trust the science” justifications for the application of power.
And being anti-science is used by the political right to allow them to make claims without evidence. Once they have reduced trust in science enough by amplifying what might be a small percentage of problems they'll be able to justify any action and if the research shows otherwise they'll just call them "elite liberal east coast ivy league etc..." to dismiss it.
Mathematical proofs can be wrong. People do make mistakes. But yes I think it’s less likely there is a mistake in a (reasonably) self-contained line of logical reasoning than a statistical study which has so many potential sources of both nefarious and accidental issues.
I agree but It’s weird that now Math is being politicized in California then. No limits to activists I guess
I am out of the loop, but surely this would be politicized math education? Which is absolutely different from mathematics research and not related to reproducibility.
how does that happen? idk much about what's going on in the US, but the what now? how do you make math political?
What happens is social scientists look at certain outcomes like people who go to Harvard are the top economic winners and people that take advanced courses go to Harvard. So the reasoning is if we can get people of an arbitrary group into Harvard we will “even out” outcomes.
But what this logic fails to consider is that people who graduate from Harvard aren’t successful simply because they went to Harvard. Their success comes from many attributes like their intelligence, etc that advanced courses are designed to separate the cream.
So then they organize and legitimize their power (removing merit and replacing with lotto or affirmation quotas) by claiming the existing system is racist. When you ask for specific examples they respond that it’s “systemic” and although no one can detect it, it’s imbued in everything. The solution is “anti-racism” which means to make up for past discrimination by systematizing present and future discrimination. This is why your HR department probably has a commissar on it now. They might call it DEI Officer or sone other bullshit job title.
This is what social sciences have contributed the last 40 years.
That's one thing that social sciences has contributed though you didn't provide any sources.
"This is what social sciences have contributed the last 40 years." This statement implies that's the entirety of what they have contributed which is false.
I mean, what I'm pointing to here is critical theory. From CRT going into gender and critical queer theory all the way into fat studies, etc. It can all be grouped under grievance studies and it's a large part of what the social sciences have outputted. It also happens to be political useful and used to cite "the science" as a justification for wielding power and making claims to knowledge.
Anyways - you seem to like to take this line all the time - "where's the evidence". It's everywhere. It isn't my job to keep you informed of the world you occupy. Either willfully or not, your inability to keep up on developments isn't an excise to demand "sources" when you have access to the same search engines as everyone else. This information isn't difficult to find.
I'm torn here.
Yes, some people use "where's the evidence" as a conversational gambit to try to shut down discussion they don't like. (And if you provide evidence, they may say "that's only one source, got any others?") And if they're being dishonest in asking, there's no point supplying the evidence they request; it is useless to try to have a conversation with those who will not listen.
On the other hand... when you make a claim, the burden of evidence is actually on you, not the other person. And if you say "you have access to the same search engines as everyone else", well, that's true. On the other hand, one person writes a post, and ten people read it, or a hundred people, or a thousand. Making the thousand do the searching, instead of having the one writer do it, is really inefficient.
This leaves you at "do the work of providing the evidence, but don't feed the dishonest trolls", which is... well, at best it's not very actionable advice.
I told you I was torn...
so the solution to racism... is being racist against different people?? what even
this has to be a joke, it makes no sense...
It isn't a joke. In fact, mainstream books such as "White Fragility" and "How to be an Anti-Racist" literally advocate for this. We see policy decisions justified by these "studies".
I mean, affirmative action is literally this in practice.
> how do you make math political?
By proposing a math curriculum that requires teaching all students the same material, regardless of their ability, with the aim of increasing social equality.
I'm OK with that aim; but I know from my own experience that trying to teach calculus to someone that's not ready for it isn't just a waste of effort, it's disastrously counter-productive (I totally fell out of love with maths when I was taught integration, failed to "get" it, and my well-regarded teacher didn't get why I didn't get it).
My understanding is that nowadays in UK state schools, maths is largely student-paced, using worksheets; they've given up on trying to get a whole class of students to all understand the same stuff. That's partly because a set of worksheets is much easier to come by than a good maths teacher, of course.
I'm not a maths teacher, and I don't know enough about the California curriculum arguments to have a view.
all students in the same grade learn the same material (atleast till tenth grade, after which they can choose their stream and now add math as an additional even in non stem sections)
Math is not politicized in California. Math education, and the social applications of math, are politicized.
I don't have much to add, only that it's been my personal experience as well. Fraud and non-reproducibility used to be such ethereal concepts to me, issues that are discussed at a systemic level as an intellectual exercise.
But after becoming a scientist, coming across so many fraudulent papers, so much non-reproducible and poorly done research. And seeing how it's affected my own, the thing that's dearest to me, having to build on top of those results and work in that global environment. It's been heart breaking, I feel no love for science anymore.
What field are you in where you are coming across lots of fraudulent papers?!
I spend most of my career reanalyzing others' data and combining it together into larger datasets, in molecular biology and genomics. The only time I encountered mistakes it was from my own labeling errors or bugs, pre-publication. And in pre-publications datasets I would sometimes detect accidental swaps on the labels of samples as part of QC checks.
Social sciences have a reproducibility problem and is littered with fraudulent data. Of course most of it isn’t really science but they (social scientists, politicians, activists) that rely on it to justify applications of power insist it is. Because otherwise it’s just anecdotal, which it mainly is.
“Sokal Squared” is a great example of the problems in the field.
Do you have evidence showing this?
I literally gave you a place to look in the post, Sokal Squared so here's a link to help you along.
https://en.wikipedia.org/wiki/Grievance_studies_affair
Regarding the reproducibility crisis:
Machine learning
But surely ML did advance a lot in recent years?
As the saying goes, science advances one funeral at a time, but it does advance.
It seems overall you can still have an optimistic point of view of science while accepting it does not work as well as it could.
That's fair but that's the systemic view of the issue. When you are in the trenches you can't look at the good and throw away the bad. You have to compare your work to the fraudulent work, spend a lot of time reading and trying to reproduce the fraudulent work, cite the fraudulent work, compete with the fraudulent work, publish as fast they do, with results better than they produce. And even the work that's not fraudulent but just bad or rushed.
It's a race to the bottom, and how can we ask that people compromise the quality of their work to compete when the only motivation for doing this work is the love of the craft. It's certainly not the money, or the environment, or the work-life balance, or the prestige.
> When you are in the trenches you can't look at the good and throw away the bad.
When in the trenches of science this is literally your job. What seems like the likely hypothesis, what model that explains the data is good or bad.
Papers are not textbooks. They are the boiling cauldron from with some bits eventually emerged as fully cooked facts.
The only way that I can square others' experience of science with my own is if they take the Discussion sections, which are free-form extrapolation about future directions, and treat them as if they were asserted as truth.
None of what you said relates to the problem I described. It relates superficially to the sentence you quoted, but only if you strip it of its context.
I'm talking about malicious, fraudulent results. Made up numbers, code obfuscated and manipulated to do things different from what's being claimed. I resent the implication that my issues stem from a naive, idealistic reading of the discussion section rather than a thorough examination of the methods.
Sorting the good from the bad is one thing, creating and evaluating models, good models or bad models is one thing. Having to compete AND CITE fraudulent work, lies, imaginary models to be able to participate in the system is not "literally" science. And it's not the job any self-respecting scientist should want to do.
I'm sorry. The idea that there is outright fraud running rampant is so foreign to me that I clearly can not interpret your experience very well.
Fraud is probably not evenly distributed.
The more money is involved, the more fraud to expect. This should not surprise: fraud goes where the money is. And where there is money, the stakes are higher.
But I have seen reprehensible behavior even in opponents of the Younger Dryas Impact Hypothesis, where practically no money seemed to be at stake.
This generalization would occur in any occupation and probably doesn't. There should be consequences except many in thread now cast doubt on the whole of "science"
For one person, a small amount of money is life changing motivation.
I would not give up hope in science. In the short term, fraud is a problem, but science is a process and in the long term it will (eventually) self-correct, though the time scales for that process may be quite long indeed.
To that end, does anyone know of any list of techniques for spotting or identifying fraud in scientific works? This is a subject that I wish somebody would write a book on, so I am curious as to what techniques people use for this purpose.
A list of resources for you to read up on this is below.
But: science isn't really self correcting. Someone has to step up and fight to correct it, every single time. Spend some time learning about this problem - as I have - and you will find the overriding attitude is one of despair. Science isn't self correcting, it's arguably self corrupting:
1. There's no incentive to pick fights with colleagues so it almost never happens
2. Even when someone does pick that fight the institutions go to ground and defend their people, so there's no outcome.
3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret.
4. And usually after that they ask the fraudster to submit a "correction" (i.e. higher quality fake). The idea that you should maybe NOT let a fraudster have a second try once caught, does not seem to occur to the brightest of sparks that run our scientific institutions. There are actually cases where this has happened and then the original complainants spotted that the newly submitted correction was also fraudulent.
How do these papers get spotted? Spend time on PubPeer. It's a site where people compare notes on dodgy papers and how they're being detected. Also read the blogs of people like Elizabeth Bik, Smut Clyde, read the old blog posts by Joe Hilgard when he was still in academia, follow https://twitter.com/steamtraen
tl;dr There's a range of techniques used for different fields, often by looking for re-used images or data across papers that shouldn't be re-using them, or by spotting internal inconsistencies in reported data tables.
"3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret."
Do all of them do this?
Yeah. Academic culture is totally broken, they institutionally assume at every level that everyone is honest and fraud doesn't exist. Therefore if someone points out problems with a paper or its data, it must be merely a mistake and the civilized thing to do is ask for a correction.
There appear to be no limits to how absurd this phenomenon can get. Here's an example:
https://pubpeer.com/publications/7611C2EEE5789028895D8BD02A1...
I still love science. I just don’t think the academic model and peer review publications makes sense anymore
> Welcome to science folks.
There is no question that a very significant portion is wrong and some of it is fraud, but I feel like this is the wrong summary takeaway. Science isn’t the reason that science is messed up, people are the reason. All fields of human endeavor suffer from the same problems due to emotional and political and selfish people. It’s more like, welcome to humanity. Science is actually the best thing we’ve got, there is no alternative that has less BS and more truth. The 10% or so of science that’s right has transformed the earth in the last century.
I worry about framing this the right way, about the subtleties of how you say it, because there is currently a war being waged on public trust in science, and to some degree that anti-science war is being won. It’s potentially damaging to say “most of science is wrong” and just stop there. That’s a misleading framing in my opinion. In order to fix the funding problems, society as a whole needs to have trust in science, to believe that the majority of people doing science are politically impartial and also not wasting money or lining their own pockets, to believe that scientific progress is human progress.
It’s important to note that the incentive problems you cite mostly aren’t caused my any malicious intent. Disagreeing with someone else’s research, in my experience, isn’t often done with the primary goal of holding back good research, it’s done because the researchers actually disagree on the science, and the reviewer actually believes the proposed paper isn’t complete or correct or up to publication standards. Moreover, for science it’s very important for researchers to be critical of each other. That is part of why we need more replication study.
I think people confuse “the process of science” for “something that is”. People think science is a thing instead of what it is - a method/process to discover objectively true things. Or at least prove what is objectively false within our shared reality.
A lot of things look like science and use the word “science” in them but don’t use the process of science to make their claims to truth. This is useful because how can you argue with “the science”?
> how can you argue with “the science”?
Yeah, totally agreed. It doesn’t even make sense to suggest that testing to check whether things work or not is the wrong process. The scientific method is tautologically true, right? This is one of the reasons I cringe at the yard signs in my neighborhood that declare “science is real”, as if it needed an affirmation. Somehow that seems to give more credibility to the idea that science might not be real, whatever that means. Of course science is real. We don’t have modern houses or cars or computers without, sillies.
I don't think you meant it this way, but I don't really like the phrasing that it's a "framing" problem and that we "need" to "trust in science" and "believe in political impartiality". It sounds like you're talking about knowing there are real problems in how we practice science, but we should manipulate the public into believing in it and trusting it anyways.
Trying to manipulate people like this is guaranteed to backfire long term. People will realize that it's being done and trust things coming from the world of science even less. Instead we need to acknowledge that there are real problems in how we practice and communicate about science and take visible steps to fix them.
I didn’t suggest people should be manipulated, you ignored my qualifying clause in order to interpret what I said negatively. What I said is that trust is required in order to fix the funding problem. Our funding process by and large does not fund things that have no public support. I did not say or even imply that we should attempt to manufacture trust. If anything, the whole problem is that right now the distrust is being actively manufactured, and we need to stop that.
I agree completely we should fix the problems, and I have no problem openly discussing what those problems are. In order to do that there needs to be hope that fixing the problems is a viable and likely-to-succeed activity. Summarizing all of science activity as broken is neither accurate nor helpful in terms of fixing the problems, right? That really is a framing problem because it is not broadly true, the parent poster implied that the practice of science is the problem, when it isn’t the problem. The problem is that people are involved, and science is actually one of the least problematic things we do. It’s incorrect to call out science as broken without comparing it to all other fields.
If everyone is convinced that science is a complete and total waste, and people are opposed to spending tax money to fund it, we will not be able to fix the existing problems. If we believe something about science specifically is broken, and not with the rest of the world, then we will come to the wrong conclusion about what is broken and be unable to fix the existing problems.
The current funding system needs a massive change at least. Funding should go to universities in general instead of specific projects. Get rid of most of the KPIs.
So that we can reinstate the old order of eminence-based science? I think the funding system should be rather be more anonymous, more global, more detached from local elites ... like universities, organization close to government/administration etc.
You mustn't forget that universities are only one player of many in the science system. And the main task of some universities rather is education and not research.
I’ve never seen KPIs in academics, is that a thing now? I’ve seen KPIs in tech companies who like to model Google.
There is a lot of funding going to universities, and a lot going to specific projects. I think maybe the main model for research funding generally is that funding doesn’t just appear, people have to go ask for it in the form of a grant proposal. Those grants typically once granted go 40% to the university anyway. Universities, for their part, have been absorbing money for larger and more expensive administrations for the last 30-40 years, and it’s out of hand. Most people I know practicing science feel the opposite of what you said - that less funding should go to the university and more directly toward funding research.
> It’s potentially damaging to say “most of science is wrong” and just stop there. That’s a misleading framing in my opinion. In order to fix the funding problems, society as a whole needs to have trust in science, to believe that the majority of people doing science are politically impartial and also not wasting money or lining their own pockets, to believe that scientific progress is human progress.
Science is losing the battle for public trust because it wants to simultaneously be an infallible source of truth and this messy, chaotic discipline where we tumble towards an approximate answer. It gets defined as a one or the other when it's convenient.
In the first breath: Oh, X% of all published papers are wrong? No big deal, that's just how science works. Can we have another 100 billion of taxpayer dollars please?
In the second breath: The Science Says vaccines are safe and effective. Take it or get fired.
You’re conflating science and politics and business. Science isn’t firing people, companies are. The science is right, the vaccines are relatively safe and effective. People whine about the mandatory actions, sometimes with good reason, and we could probably handle this better, but ultimately why should you be allowed to spread a disease at work that has an unfortunately high probability of killing people (or even of just causing someone else to be out sick for a week or two)?
The US currently somewhat operates on the principle that companies are allowed to make and enforce their own rules, as long as they’re legal. And there’s a long list of things you must do if you don’t want to get fired, including work forty hours a week, and not be a jerk to your co-workers. Do you think companies should not be allowed to fire people who choose to do things that can hurt the company’s bottom line? Do you want the government to monitor businesses more than it does today?
Science funding is complicated and political, but the high level summary of the situation is that the public funds science right now because that’s what the public wants, because a good chunk of society understands that doing science has proven to be a great investment. It would be a bummer if the public decided not to fund science or education, that sounds like the fastest way for the US to become irrelevant globally.
Anyway, what is the alternative to doing science? Do you like your computer/car/cell phone? Do you want to see technological progress stop? Can you name a better approach to seeking truth than the scientific method? What has ever proven to be more effective than science?
For everyone reading, and appropriately outraged, I challenge you to put your money where your mouth is.
Even $5 to someone with a long history of finding scientific fraud puts you on the right side of history: https://www.patreon.com/elisabethbik
I’ve been donating for years and I’ve never been more proud of that decision.
In biology and some parts of chemistry, I would say yes 80% of papers have a wrong conclusion (experiments are ok but not interpreted correctly) or completely wrong. But in other fields it is not the case. I'm pretty sure mathematics are much better. Physics it will depend on which branch some are closer to science-fiction anyway so are not verifiable, but most physics is good quality. It is hard to consider science as a whole because it is absolutely not an homogenous lump and practices in reviewing, publishing and peer pressure are radically different.
We should introduce a maximum number of papers per researcher per period, like one per year or one per quarter.
That way the only possible competition will be number of citations. The pressure will be for quality, not quantity.
And you believe that won't be rigged? Sweet summerchild. The pressure will be for citation, crudely said: citation bait. Well researched articles on researchable topics aren't quoted that much (it's boring, and what's left to investigate and publish about, after all?). I don't think a single measure is capable of correcting the current system.
> The pressure will be for quality, not quantity.
Well, popularity, anyway. While popularity is sometimes a useful proxy for quality, that's not always the case.
Probably the best proxy for quality is popularity that endures for centuries since we can’t measure quality directly. Either way, if there’s a max of one paper per year, scientists will be trying for the most impactful work.
People are already boosting their impact scores by requiring papers they review to add citations for their past work.
Performance Metrics always result in people gaming the system, and impact scores have quickly become unreliable.
At the same time, reviewers are chosen based on their expertise in a field, and one of the things reviewers look for is that new papers cite relevant prior research. Are you suggesting that reviewers should just let papers go through without citing relevant prior research if it is their own research? Does it actually matter whose research it is as long as it is relevant prior research?
Of course it matters, adding citations to already written research is deceptive and thus actively harmful.
The point of citations isn’t simply an index of prior research it’s there to inform readers of the writers context.
When a paper is rejected because it doesn't contain references to relevant prior research, the citations shouldn't simply be added; the relevant prior research that wasn't considered should thus be considered, and the paper should be reworked with that new context in mind.
Research should be put into the proper context in the field, not based on what the writer personally looked at. If a researcher's context is myopic enough not to include relevant prior work known or done by reviewers, then the research itself is of questionable value. If someone submits work that purports to be novel, yet they haven't done a proper literature review to include relevant studies, I'd say that is itself deceptive and more actively harmful to the field as a whole.
Suppose an experiment takes 5 years, nothing written after the experiment begin should have any barring on what was done and the resulting data.
The same is true to a lesser extent for any research. Now if the field has discovered some flaw in what was done then things go well beyond the simple need for a citation.
That's not how competent research is done. You don't go into a hole for 5 years and then come back into the world with a fully-formed paper. While you're doing that research, the world moves on, and if your citations are from 5 years ago, any publication will ask the extremely appropriate question "How does this matter w.r.t. the research published since you started this work?", and send the paper back for rewrite. Or, it may ask the even more appropriate question "Why does your research matter given the results of papers x,y,z which are directly related to your research question?" Any competent researcher should preempt in their literature review, and if they can't then it's not research but instead PR.
And this doesn't even touch upon the fact that research that's going to take 5 years is going to be well-funded, and you'll have to write interim reports over that period. When you get 5 years of funding, you don't get that all at once. During that time, you'll have to answer for advances in the field and justify why the research is still worth doing.
Experiments that take a long time aren’t always expensive, sometimes you just need time. What actually happens to a sample held at X condition for some period of time?
As to other field moving on, it can be a reason to reject the paper in it’s entirety but has zero impact on the results which is the only thing that actually matters. Papers aren’t textbooks, the goal is to communicate what was done and what happened not inform the reader of some wider context.
In the end if I am reading 30 papers on some topic I really want them to be as short and clear as possible. And for topics I keep up with I sure don’t need this fluff.
> Papers aren’t textbooks, the goal is to communicate what was done and what happened not inform the reader of some wider context.
And a literature review is part of what was done, always. If what you said were true, there would be no point in a literature review at all. Instead, every paper includes one because researchers understand it's an intrinsic part of research. That's what makes it research instead of just search: research is a systematic process that involves, in every definition of the process I've ever come across, a literature review. I challenge you to find an example of peer reviewed research that doesn't contain one.
Here’s a translation of a rather fun counterexample.
https://cdn2.hubspot.net/hubfs/232514/Einstein%20E=mc2%20(pp...
But, I don’t disagree that someone should keep up with the literature. Including it as part of a paper is what I disagree with.
It’s unlikely this paper was peer reviewed. The peer review process was much different in Einstein’s time. It wouldn’t pass for research today. I kind of thought we were talking about how research is conducted in modern times, not how they did it over 100 years ago.
If you want a more recent example. How about 1966 with the counter example of Euler’s Conjecture on Sums of Like Powers.
A direct search on the CDC 6600 yielded 27^5 + 84^5 + 110^5 + 133^5 = 144^5
as the smallest instance in which four fifth powers sum to a fifth power. This is a counterexample to a conjecture by Euler [1] that at least n nth powers are required to sum to an nth power, 1 > 2.
And that’s the paper. They could have gone into great detail about the program used etc, but it clearly communicates what was done and the results so why bother.
“I consulted a magic box and arrived at the answer” is precisely the kind of paper that is an example of search, and not research. I mean, they plainly stated that’s what they’ve done.
How would you go about redoing exactly what Lander and Parkin did? You can’t because they don’t really say. Isn’t this whole thread about a lack of reproducible studies resulting in fraud?
But anyway, I like what you’re doing here because you’re beginning to document why the modern peer review system was developed in the first place in the 70s. Sounds like a good topic for a research paper.
A counter example is a proof unto it’s self there is no need to derive it again, just verify the counter example is correct.
Many modern papers are of this form. If you have some specific data as recent enough to be valid I could provide one but I am not going to just pick a paper from 1990 and you say nope not recent enough.
There are many reasons to reproduce papers, it doesn’t matter if the author is unquestionably correct. One is to establish that they actually did what they said they did. What if the authors stole this result? It’s certainly happened before. If we don’t accept students divining solutions, why should we accept that from scholars?
Another is for posterity and for future generations. If I give the paper you cited to a student, they can be wowed at its brevity and impressed by its genius, but they learn nothing.
I get you’re trying to prove your point but all you’ve done is solidify my position in my mind that literature reviews are essential to good research. You’ve given me a great topic for my talk to our incoming class of Ph.D. students in a couple weeks, so thanks for that, I mean it. Cheers.
So you aren’t actually open to new information, that’s disappointing but unsurprising. I get your stance but again papers aren’t textbooks.
O well, there’s a reason science advances one retirement at a time.
Sheesh, I don’t think that was called for. Was a good discussion until that :/
Yea, I was going to delete it.
Just got annoyed you tried to weasel out of picking a date and tried to pretend you had won the argument.
You get what you incentivize, and this is what we incentivize.
It’s probably about the worst in any field dominated by studies and statistics. Harder sciences are harder to fake as bad results can be more conclusively falsified.
Yes, it’s great when we rediscover some nugget of wisdom that everyone has already explained.
It’s quite useful to listen to Buffet and Munger. They’ve accumulated lots of wisdom.
“You get what you reward for” https://youtu.be/hJYLJRr3hEY
This hour of Munger is well worth it:
Here’s the transcript:
I think any parent or manager also knows this…
Even in the hard sciences, the results are fundamentally about statistics. And they often require very expensive machines for reproduction.
I'm not sure if it is a good idea to try to optimize the scientific process for fraud resilience. In the end, we still rely on people doing the right thing most of the time.
I have a friend with degree and Ivy, got phd, and did post doc.
He said something exactly the same, and his conscious wouldn't allow him to work in the field of science.
He left the field many years ago. He delivers mails now.
What is the field of science
similar to the field of dreams but motivated by money instead of altruism
So capitalism?
I dunno, I feel like it’s bad but getting better. 20 years ago, nobody tried to replicate anything. Now it’s much more common and, I hope, expected for results like this one.
80% of science? How would things like GPS or rockets work if that was the case? What we consider reliable technology would be blowing up everywhere around us constantly.
Science, not engineering
Where do you think engineers get their equations?
For the science part of it: >100 years old scientific literature
Possibly this was sarcasm, but I’ll bite just in case. There is quite a lot of scientific research that informs modern aerospace engineering: https://phys.org/tags/aerospace+engineering/
You’ll find similar physics research for example in nuclear power, optics, quantum computers, telecommunications, engines.. all kinds of engineering really. Chemistry is also quite relevant and there are new, stronger materials being engineered all the time based on research.
It’s definitely not all just 1900s physics driving everything, even if you say that for some reason anything researched then doesn’t fall under the 80% of science that doesn’t work (which GP didn’t).
Where do you think scientists get their data?
Much of what you think of as science came from engineers.
Is rocket science possible sans science? Can you be more specific what engineering without dependency on any science is? Like how do you even approach inventing gps as an engineer if 80% of physics calculations don’t actually work for example?
Almost all of what is called "rocket science" is firmly in the engineering wheelhouse.
GPS relies very little on 20th century science, beyond GR corrections.
The science behind orbital mechanics is just Newton's Laws, stuff thoroughly settled centuries ago.
That is Engineering.
Do you also consider stuff like physics and chemistry engineering and not science? Otherwise I’m not sure how you’d achieve anything as an engineer if 80% of your scientific understanding of how your materials work were wrong, for example.
Some is, some isn't. Much of what we call science was work done by engineers.
So going back to the original topic, 80% of science doesn’t work but actually a lot of science is done by engineers and that’s what the 20% that works is?
Got no grounds for opinion about physics or chemistry. Certainly some of it is good work, although I harbor doubts about the ultimate usefulness of string theory.
But your example of GPS, and others' of "rocket science", show that you have even shakier ground for opinion, along with insufficient respect for engineering. You are not alone in that: scientists get good publicity. They even get dedicated press offices at universities.
Engineering is great. I guess it just wasn’t clear to me that GP didn’t mean science older than 100 years or that physics and chemistry didn’t fully count as science when I replied to their blanket statement that 80% or 90% of science was wrong. If you define science to exclude anything that is in practical use by engineers and/or has become “settled” over time then the statement is less surprising (but also perhaps less provocative and interesting).
Science is about discovering models that account for observed nature.
Engineering is about making things that work. GPS is very firmly on the side of a thing that works. Likewise, rockets. People sometimes use those when doing science.
So your trotting GPS out as an example of correct science did not in any way illustrate what you imagined.
I mentioned GPS because it depends on general relativity (or at least I believe it does).
Are particle accelerators a better example for you, in that they wouldn’t actually work if 80-90% of science is wrong? Or is particle physics also actually engineering and nothing to do with science?
Anyway, my guess at this point is that we both agree on lots of the same technologies that actually work, just not on if general relativity or physics or chemistry are technically science or engineering, and therefore whether they would fall under the “90% of science doesn’t work” statement. Its hard for me to imagine engineering without a dependency on scientific knowledge, but it’s interesting that that’s a valid point of view out there.
GPS would, in fact, work totally fine if the effects of general relativity were exactly zero. The software does, however, need to apply adjustments to account for effects of GR, in order to produce precisely accurate results.
A particle accelerator is a tool used for conducting science experiments. That they work, for that purpose, is purely a product of engineering.
At issue is not "whether science works". It is, rather, what percentage of published papers in science journals are crap, and advance the field not at all. The people in the best position to know say they are appalled at how many are.
Would they all say 80%+? Obviously not. Do any? Absolutely.
The specific statement I was responding to was this:
> I can say I believe 80% of science wrong AND not reproducible, 10% being an outright fraud (10% being legit).
.. which I interpreted to be a statement on "whether science works."
If we instead constrain that to 80% of science that was posted recently in journals, and mainly not in hard sciences, then I could totally get behind that. But I don't think that's what they said, and I think that's why I've been confused by this whole thread.
Don't think GPS would be possible in this day and age, I mean, "inventing" it and "implementing" it from zero. It was possible to "invent" and "implement" it in the late '70s - the '80s because back then the "capitalist virus" hadn't entirely invaded the US MIC (and the scientists associated with it), it was all basically a socialist system. That is no longer the case, even when it comes to Government-sponsored work.
What about Starlink?
I don’t think it’s remotely accurate to describe GPS as “socialist”. It was a military project. There’s a long list of military projects with more funding than GPS that have been invented and implemented since then.
That’s what I was saying, the military used to be socialist, even in the capitalist US. It was definitely socialist during WW2. I think it started becoming real capitalist once the (first) Cold War ended, I think the contractors thing started becoming more and more important throughout the ‘90s and the wars in both Afghanistan and Iraq were fought by a capitalist US military. It’s telling that these 20 years of war (2001 - 2021) didn’t bring us almost anything worthy when it comes to science funded by the military, unlike almost all the other wars in which the US had been involved.
Starlink is mostly a marketing thing, and is mostly based on science that had been funded during the Cold War.
I don't know what you mean. The military isn't an economic system, and it's definitely not socialist. It has a rigid authoritarian hierarchy, and it's funded by taxes, always has been.
The years you cite are when drone tech was developed and deployed in combat, and cryptography improved. The DoD funds all kinds of research through the university system that isn't public, you may have no idea what it brought us because it's too recent. The DoD is funding all kinds of projects today, despite our capitalist tendencies, for example: https://www.defense.gov/News/Releases/Release/Article/309821...
I don't know what you mean about Starlink being marketing. They've deployed more satellites than all other satellites to date combined. You said you thought a project this size wasn't doable today, yet there it is. It seems like you're arbitrarily judging what projects you believe are worthy of consideration in order to support a preconceived notion. Why do you believe GPS involved more invention or development than Starlink, drones, cryptography, machine learning, or anything else? GPS technology was also developed before the US Navstar project began, the science already existed and it wasn't some kind of lone independent invention.
> The military isn't an economic system,
The system of which the military is a major pillar is definitely also an economic system, and the fact that a major part of that system (the military "sub-system", in this instance) was run in a way that was antithetic to the stated ideology of that "all encompassing" system as a whole ("capitalism", in this instance) was quite interesting.
I'll give you drones, even though the drones that have had most of an impact in the present war are the consumer-oriented ones, which are not directly the work of any military system (I'm talking about the small drones that help with artillery targeting). The Bayraktars were quite rapidly neutralised by the Russians once they got the hang of it, and at this point they're merely a PR thing.
Cryptography is also a result of Cold War funding, in fact it's one of the quintessential results of that war when it comes to our industry.
> Why do you believe GPS involved more invention or development than Starlink, drones, cryptography, machine learning, or anything else?
Didn't say that, quite the contrary, most of those are the result of direct and sustained Government funding via the military, i.e. what I called socialism. Again, Starlink would have remained a pipe-dream without Reagan's 1980s Star Wars Program.
Anyway, I'm happy for the current status-quo, i.e. for the best and brightest in our industry golden hand-cuffed at FAANG companies, spending their best years worrying about their comps and about KPIs and OKRs, had they been part of a Manhattan-like or Apollo-like Government-run system then things would have been much, much, much worse (in terms of peace on this planet, that is).
I agree with everything on your list. Most of these problems are quite difficult to solve, but I think (2) is doable if we can convince politicians to unfreeze science spending that has been stagnant for decades. We need a new set of early career awards that require 50% of time/effort to be spent on reproduction studies.
> We need a new set of early career awards that require 50% of time/effort to be spent on reproduction studies.
Then you also need a realignment of the entire academic system as well. A new professor spending 50% of their time redoing the experiments of others, especially on capital expenditures and grad student time is going to be at a severe disadvantage against others who spent 100% of their time building their own research agenda.
We'd probably need a whole new degree that's like a PhD, but whose dissertation isn't a novel extension of the field, but reproduction of other studies. Because as it is, a grad student can spend up to a decade on a Ph.D. just doing novel research. Now you want them to spend 50% of their time on reproduction? So is a Ph.D. supposed to take up to 20 years with your proposal? Or does a dissertation contain 50% less novel research?
Certainly we should have funding for reproduction studies, but it's not enough to just have a grant; you have to have a valid career path doing such work. That doesn't exist today.
> We'd probably need a whole new degree that's like a PhD, but whose dissertation isn't a novel extension of the field, but reproduction of other studies.
I don't think that would work - those positions would go empty. People apply for grad school because they want to do original research. On the other hand, once people are already in the academic pipeline and they run into struggles advancing from postdoc to assistant prof or maintaining funding as an assistant prof, I think a lot of people would take a 50/50 grant if lets them keep one foot in the door for original research. They aren't going to turn their noses up at the only grants they are competitive for.
You do raise a good point that labs with 50/50 grants wouldn't have as good of a value proposition for prospective grad students as labs doing 100% original research. I'm not sure what the solution is there.
> They aren't going to turn their noses up at the only grants they are competitive for.
They would if it doesn't make them competitive when it comes time for tenure review. Reproducing research doesn't get one tenure, so accepting such grants would mean tacitly admitting you should also start looking for a new position in a couple years when you go up for review. That's why I said we need an realignment of incentives across academia in addition to these grants, which would give a career path to people. Maybe it's worth it to create positions for people who just reproduce others' research, I don't know.
https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in...
Humans are not incentive-following automata, though we have incentive-following tendencies.
would be nice if people kept this in mind before they crucify others for not "trusting the science"
100% agreed. I can't seem to get people to understand - most research studies find results that are not reproducible. Putting it nicely "Research has a reproducibility problem."
> (1) As a researcher you lose money if you don’t produce
This is the biggest problem in my view. My work in R&D taught me that most of the time we don't produce anything. It's high risk. But it's high in rewards, often in adjacent areas not primarily the focus of the initial brief.
Surely all serious investors understand this. Research is something we do for marginal returns. It's not an "innovation factory". With things currently stacked against risk, research can only yield tepid results.
Investors are making solar farms be sited in deserts, are pouring money into fusion startups, and bid up fraudulent Energy Vault (NRGV) to $2B. Investors canonically do not understand.
> Surely all serious investors understand this.
Not necessarily when the investor is an academic institution or a government.
By "investor" I was thinking of people who risk their own money.
One of the major root causes of the problems with the system.
Ad 5: more data doesn't help if the main problem of your research is that your data is shit. People benefit from generating tons of shity data because they get more money, which looks good on their resume. If their results are shit, they simply write "more data is needed" and they benefit even more from the call for more data.
The thing is the difference between “Science” provable hypothesis, challenges and the current research industry.
It would appear that this isn’t really “Science” if the ideal of research is being veered away from so much.
Vested interests suffocating the process.
Wait, doesn’t this mean that funding sources can impact what “science” gets settled?
A more cynical take (or maybe realistic?) is that science becomes "settled" once the revenue streams are established.
Which would also imply a total lack of funding for counter-narrative research.
There's probably a higher percentage if the article comes from the journal organic syntheses
Yes indeed. The sexy world of academia. Fast cars and easy money.
First off science isn't a section of society, company, or group of people. Second 80%? Your wild hyperbole has no value, makes no point, and reeks of immaturity