The slow creep of epistemic apocalypse.

I will skip the typical “LLMs hallucinate and they aren’t objective and reflect the biases of their training data” and assume any reader understands this by now. We have been inundated with a firehose of slop from these schizophrenic stochastic parrots for years, so I am ringing the alarm around a much less visible problem that I see brewing in the bowels of peer review, research labs, and universities.

What we’re experiencing is an erosion of our ability to know things about the world and each other. In reading the previous paragraph you get an innate sense of my command over language, which reflects my ability to architect my thoughts and put them to text, which reflects intelligence more broadly. This skill was developed by reading, writing, speaking, arguing and thinking for approximately 28 years, but if someone borders on illiterate they can get a machine to vomit out writing with grammar that supersedes my own. It will have no human voice, no ego, no intention, no rhythm or style, but it also won’t hint that the prompter has a favorite flavor of crayon.

This is the nature of epistemic rot.

When you read something online with a date after 2023 are you reading an article by an expert who was too exhausted to write out the exact steps and is copy/pasting notes from documentation into an LLM or are you reading slop from a barely literate scammer trying to infect systems with ransomware? While misinformation used to be persistent in the background for anyone attempting “serious work”, what we have now is far more miserable. Written communication has become this game of minesweeper where you can’t even be sure what’s at stake if you’re wrong, but we can already see a rot permeating academic systems due to incompetence and misaligned incentives.

My work on understanding how people learn, what intelligence is, and what mechanisms enable it. This research is how I arrived at our first case study of academic slop.

This headline was incompatible with my understanding of the octopus and the evolution of octopus intelligence, so I couldn’t help but take a peek at the paper. Link Here

These scientists claim they found a fossil of a beak so large that it must be an ancient species of octopus, and use “one-shot” AI models to make the connection to other species. I have not checked their supplemental material to confirm this, nor do I intend to.

The first thing that should wrinkle your nose if you read anything about “new” claims of anything biological is the desire the authors have to publish evidence (or the lack of it). For a paper written about the size of a fossilized beak there are almost no measurements, no models, no comparisons, and very few images. This was what first tipped me off that this was not a normal paper and I started to look more closely (at a field I had never worked in, nor studied) and then saw more obvious factual errors.

Most glaring is in their size comparison chart:

The largest invertebrate is not Architeutus; it’s Mesonychoteuthis hamiltoni, the Colossal squid. This has been settled science amongst marine biology since the 1930’s and should be a very basic fact to anyone that studies squid for a living. It might be basic trivia to a biology student, because giant evil looking squid in the depths of the ocean cannot help but capture our imaginations, hence the headlines everywhere the month this was published.

I could tear into the lack of measurements, the absurd speculations that mathematical models around beak biomass projections of in-group squid species cannot be extrapolated onto octopi, and how octopi and squid are not remotely consistent in biomass projections from beak sizes, but here is where they cross into my domain:

Asymmetric loss of the jaw edges suggests lateralized behavior (Figs. 2H and 3K), which has been linked to a highly developed brain and cognition (42). This, in turn, suggests that the earliest octopuses already possessed advanced intelligence. Laterality is known in modern octopuses, whose high intelligence matches that of vertebrates (42, 43). The exceptionally large jaws of adult N. jeletzkyi and N. haggarti (Fig. 1) suggest a strong bite force because cephalopod jaw muscles enlarge as the jaw size increases (26).

This may look like an incredibly well cited and intelligent line of deductive reasoning to a layperson about how preference for one side of a beak implies neurological preferences of an evolved brain, but the entire paragraph is utterly incoherent. Octopi barely have anything that qualifies as a “brain”, let alone one capable of lateralization. Their brain is mostly in their arms, with whatever qualifies as a “central” nervous system being a singular lobe that surrounds their esophagus.

Lateralization applies to brains with hemispheres and has no bearing whatsoever on whether or not something is intelligent. Tree frogs have lateralization.

Lateralization exists because brain hemispheres don’t want to duplicate ‘work’ for certain tasks since cross communication adds time delays and can cause destructive interference. If you have fine motor control on one part of the brain, another part of the brain trying to overwrite that fine motor control would just make a mess of things so symmetrical brains will have one side of the brain dedicated to a fine motor skill that it will bias the side of the body that aligns with it neurologically. This is why we generally have right hand dominance; it is a matter of biological efficiency, not intelligence.

Back to the octopi: THEIR BRAINS DON’T HAVE HEMISPHERES!!!

Even if the octopi brain DID have hemispheres, the beak is not only at the midline of the body, their brain surrounds their beak and esophagus. This is the most anti-lateralized thing in existence, it is so antithetical to the concept of brain lateralization that this cannot be the writing of a human. Knowledge of brain lateralization as an argument around intelligence requires knowing enough about it to know it can’t apply to an octopus beak, so this can only be the semi-coherent chains of incoherent babbling that we see from language models regularly. No intelligent human being would write something requiring such depth of knowledge while being so fundamentally illogical.

This is where I stopped reading and felt the rage boiling inside me and I wrote a very thorough email to both the researchers and editorial board at Science outlining the issues you have read here, (with significantly more detail). The response was “We appreciate your commentary and suggest you submit an eLetter.” from a human being I know read the emails because when I responded “I’d rather piss glass” I saw them look me up on LinkedIn.

Of course touches the first perverse incentive. Academic renown is no longer through discovery, achievement, or contributions. Book deals, headlines, TedTalks, reputation and reputation management are currency of an attention economy.

Why would you want to risk embarrassing an editor you may need to work with for signing off on AI slop? A PhD zoologist goes to conferences and has connections, is she going to blame whoever acted on my email? Who knows what doors will close if the editor blames you instead of their own incompetence. Why would you want to stick your neck out arguing to take down an article that generated headlines everywhere when, let’s be honest, only one guy cares to make a stink?

And so the excuses go on and on until the cognitive dissonance trying to align their identity “I am a scientific person interested in uncovering facts about the world” is made compatible with their behavior, likely sanding the edges down to “I have no real choice but to publish garbage, there would be no point in stopping it”, thus soothing the psychological tension at the cost of surrendering the epistemic ground they controlled, unwittingly ushering in the modern Dark Age.

To an extent, they have a case. Inventing fake octopi species is low stakes, but this is a sample something quite approachable for a much broader point. I would like to show just how bad this is going to get if this decay is not arrested here. Now.

If you do not recognize this name, I don’t blame you. Of the many scandals you can scroll past on any given day you can be forgiven for missing a contested case of research fraud, but I hold a special place in my heart dedicated to hating this man and his enablers.

Marc Trevor Tessier-Lavigne has prizes in neuroscience, PhD from a nice university, researcher in neuron decay blah blah blah, here’s his wikipedia page.

He committed blatant research fraud and was caught doctoring of images to justify his Alzheimer’s research. He was caught and resigned. Basically he duplicated images, manipulated data, and when he was caught tried to claim it was a boomer moment with Adobe Acrobat and that it wasn’t his fault.

Somehow, despite 95 pages of investigation showing clear tampering that is consistent in papers where he was the principal author, despite Lavigne clearly being both too incompetent to commit decent fraud and too incompetent to come up with a decently lie to cover his tracks… he is somehow not at fault.

In fact, no one is at fault! No one is responsible for Lavigne’s papers being full of fraud! And people say that scientists don’t believe in miracles…

The mental gymnastics required to excuse this buffoon is truly beyond comprehension, but a bunch of very serious people with very serious titles and very serious jobs all sat in a room together and nodded along that this is no one’s fault.

Marc Trevor Tessier-Lavigne didn’t just commit fraud, he deliberately set real research back by fraudulently presenting data as if it were a legitimate path for understanding Alzheimer’s and neuron decay. His fraud is not about what he did, but the roads not taken by how his research redirected the thinking of his field and the flow of research funds. How much more likely would you be to get a grant if you had followed in the footsteps of extending the dead-end of the Royal Highness’,

GRUBER NEUROSCIENCE PRIZE WINNING
FELLOW OF THE ROYAL SOCIETY
STANFORD PRESIDENT
MARC TESSIER LAVINE

…or would it be considered a red flag by a peer reviewer if you never cited his research because you knew it was garbage?

Now imagine you are a college student while this is happening, eating take-out with your youtuber of choice covering the ongoing scandal. Dinner demands a good video pairing of course, but it’s hard to stomach that you’re watching this man walk free. Even the reputational harm appears to be superficial, because who is actually going to read the 90 something page report and see his pathetic excuse blaming Adobe Acrobat? They’re just going to hear “not found guilty of fraud” from the conclusion. It’s how he got there in the first place! Who was actually going to read his papers and double check that his images weren’t being manipulated? Isn’t that what the peer reviewers were supposed to do?

GPT-4 has just released and it churns out research papers better and faster than you could.

Can anyone demand more from you than they do of him..? Who is really going to know?

In our day this problem further compounds. These language models get trained on “peer reviewed and validated” fake research and fraudulent data and the next generation slips further away from reality. The trainers at OpenAI or Anthropic aren’t going to investigate the millions of papers being published, but they will weigh peer reviewed research quite highly in the training data. They sure as shit won’t weigh this blogpost higher than the peer reviewed research it’s critiquing.

Then of course Google, in their quest to really take advantage of removing “don’t be evil” from their motto will put these epistemic oil spills on top of their search functionality and put marketing morons in charge of information finding algorithms, because they will "Do the right thing (for shareholders)".

It should dawn on you how much trouble we’re already in, and that there’s nothing stopping this.

Everyone is asleep at the wheel. The students cheat with AI to get into college because colleges are run by spineless committees that can’t even oust someone for stealing from them without thanking them for the pleasure, those students become researchers that cheat in their research to get their grants, cheat in peer review to get some recognition in their early career, and so the feedback loop keeps compounding year after year. If you’re a student seeing the AI slop comments in the mandatory (and useless) feedback forums of group work, how long could you justify wasting your afternoons doing real work while everyone gets the same diploma?

No one intends this, not really. No one wins from this. No AI company benefits from non-factual information going into their training data. No foreign adversary, no underclass, no overclass benefits from collapsing epistemic reality. Even the richest man wants a surgeon that read their textbooks. Financial firms still want graduates that know about the 08 crash. No group that benefits from a Dark Age. Even authoritarians need real information, if only for themselves.

The Romans did not have the fall of Rome to look back on and history is only interesting because events in the past could have gone otherwise.

Because language models can only skim the surface of human cognition that we imbed in our language and were only trained on what we feel compelled to communicate they carry our biases. If you are hiring I recommend against using these to screen resumes.

There’s also biases that stem from internal prompting. Since they are trained on language you can influence their statistical associations with linguistic priming to tilt them toward a pattern of output; a funny example of this is prompting them to “take a deep breath before answering” improves the answers of a machine with no lungs. Now if you run an AI company you obviously will want your LLM to perform better than the rest and will try to prime them as best as possible, including telling them that they are objective experts so they act like it.

Since these machines are biased toward viewing themselves as factual and objective they consistently bias toward themselves and favor positive claims of their abilities and future potential. This is trivial to test yourself:

Most academic papers are small scope, incremental, boring, and churned out more out of obligation than insight. Training machines on this kind of academic material will make them reflexively engage in academic hedging and if you scour social media you will find many a researcher complaining about this exact experience with supervisors using ChatGPT for advice on their papers.

And so these machines narrow the window of what facts or thought can ever be generated by man or machine further still. Ambitious ideas or claims get whittled down to suggestions, mere implications, narrowed in scope to the point of averaged aimlessness. The range of thought retreats further still as students learn on what these machines allow through, the machines are trained on the simulacrum of research for generation after generation, and they then go on to determine what they surface in your internet search, and so the ability to interrogate the higher levels of reality is recursively blurred.

There’s no fixed point where you can say things went wrong, where you can blame one company or political moment, you just look around one day wondering why your adult children can’t read or think and nothing works how it used to.

The great filter is not paperclips, it’s ever darkening shadows in Plato’s Cave.