Page by SE Gyges | @segyges.bsky.social

I'm back, I missed you all. Since this is an important subject I know too much about, here's a 🧵 explaining that AI Safety Is Mostly A Sex Cult. I don't think these people should make policy. (1/?) (alternative title: Time For Some Cult Theory) There has been a great effort over many years to distance AI Safety discussion and policy from its origins and intellectual center, because its center is a sex cult started by a fan fiction author. If you want to be taken seriously you have to hide that. This is Eliezer Yudkowsky ("Yud"). We see him here meeting with Bernie Sanders to advise him on AI Policy and arguing about AI Policy in The AI Doc, now on Netflix.

This is very senior OpenAI employee and unofficial OpenAI spokesman Roon. Roon's opinion of Yud is that 1) he is in and/or running a sex cult and 2) his opinions about AI are very important and should be taken seriously.

Yud is the person who connected Shane Legg and Demis Hassabis to Peter Thiel so that they could secure funding to start DeepMind, which is now Google's AI research division. Here's Sam Altman's opinion of Yud.

This is the schedule for the Future of Life Institute's conference on AI in Puerto Rico in January, 2015. Elon Musk would found OpenAI that December. Yudkowsky claims that somehow events at this conference caused that, and that he's upset about it, but this seems unlikely.

Dario Amodei, current CEO of Anthropic, is here in 2013 identified as a "member of the Effective Altruism community" on the blog of Yud's non-profit. He argues with yud about the "paperclip maximizer".

Anthropic papers still cite Yud on occasion, and the entire way we talk about AI uses words his non-profit popularized. "AI Alignment" was popularized by Yudkowsky's non-profit, which was the first venue to use this exact phrasing consistently.

It is very hard to discuss AI at all without implicitly invoking Yud's ideas or the rat/ea scene. Coxon, who recently left Anthropic and positioned himself as a whistleblower, screenshotted here.

I almost forgot this part: Here's Eliezer Yudkowsky in Time Magazine in 2023, advocating in favor of risking nuclear war over his hobby subject.

The guy who tried to firebomb Sam Altman's house cited Yudkowsky as his primary inspiration. I have a recording of him talking about this laying around somewhere if anyone cares, but let's be real, who else is going to ineptly firebomb Sam Altman's house? I promise I will come back to the "sex cult" part, but that's actually the easy part to explain because they post about it constantly. What's difficult is getting past the extensive, complicated and well-funded wall of denial that this is the main event. This picture isn't really a joke, though

Yudkowsky's followers call themselves "the rationality community" these days, externally they are called "rationalists", the followers "rationalism". Apparently they think this sounds less cultlike. I personally find the very long "we are not a religion explanation" either funny or insulting.

Oh, just so I have cashed this check: The fan fiction that launched a thousand cults is Harry Potter and the Methods of Rationality. It is possibly the highest-impact single modern religious text, and its influence may not yet have peaked. Silly timeline. en.wikipedia.org

Allegedly, "Effective Altruism" is a different thing and doesn't take marching orders from Yud. This is basically bullshit. The ideas and people heavily overlap, especially in San Francisco. I am also told that Anthropic is not an Effective Altruist company. This is 100% bullshit. That video with Bernie and Yud? Also in the room: Jeffrey Ladish (Palisade Research, Coefficient funded) and Daniel Kokotajlo (AI 2027 author, funded by Survival and Flourishing Fund). All "effective altruists", not "rationalists", but they all got in on that together. Where there is any separation between the two things, "rationalism" is team Yud and "effective altruism" is currently team Dario, and they're competing for attention and funding in that same space. The way they do this is extremely funny.

Yudkowsky's most viral idea, the one that infects the most people and spreads the farthest, is that AI, if we continue to develop it, will inevitably 1) achieve superintelligence ("RSI"), 2) escape human control 2) kill everyone ("paperclip them", "doom") All arguments about AI Safety and AI Alignment are primarily about mitigating this exact scenario under his exact terms. Significant modification is relatively rare. In the broad subculture, this is implicitly what "safety" and "alignment" mean.

People who think they're Effective Altruist and not Rationalist, but who are deeply concerned with "AI Safety", are generally implicitly or explicitly concerned with Yudkowsky's basic framework for loss of control and doom. This idea seems to be the most infectious because it activates the same neurosis that every apocalyptic or millenarian cult does. The world is going to end, and only I and my special group of people in the know can maybe stop it. (Also, you should give me money and have sex with me.) You can argue with them in any way you like but it barely matters. It's not, mostly, something people believe for either rational or empirical reasons, and they will simply go back to complaining about the apocalypse as soon as you stop paying attention. In person they are gathered mostly in various communes in Berkeley, California. They call their communes "group houses", maybe to make it sound less cultish. Most of the details of this are what you expect from "a commune in Berkeley". Everyone is polyamorous, everyone attends the same events, everyone lives in group homes together, everyone venerates the same few authors, and everyone is very worried about AI doom. They are all very contrarian together. Lots and lots of them work at AI companies and non-profits. The safety, red teaming and alignment teams seem especially afflicted, because that's the area you go on to worry about and try to work against AI doom, which you are sure is at least medium likely. The crown jewel of this whole thing, though, is Lighthaven. It's the former Rose Garden Inn in Berkeley, and it was bought by one of the rationalist charities in late 2022 with SBF's money, which was looted from FTX for them just before it collapsed. There are maybe five people who could have physically pushed through wiring money to a rationalist charity from FTX when it was collapsing, and one of them is Leopold Aschenbrenner. Leopold married one of the others, and she's Chief of Staff at Anthropic. Leopold is, allegedly, an effective altruist and not a rationalist, but you can see how this gets very difficult to keep straight. In what year were you one and not the other? Was your organization, at the time, financially tied directly to Yudkowsky or no? Lighthaven hosts the local Astral Codex Ten reading group with Scott Alexander, a reading group for The Sequences, which is essentially the Rationalist Old Testament, a conference for AI people called The Curve, an EA conference called EAGxBerkeley, MATS, aka "Machine Alignment, Transparency, and Security or ML Alignment & Theory Scholars"

and SlutCon, organized by Aella. Gretta, pictured here, is Eliezer Yudkowsky's primary partner, and draws a salary of about 200k from his nonprofit as his assistant while also holding this event at LightHaven. She has no prior qualifications for any job at his nonprofit

One hopes no illegal sex work happened at SlutCon, and the presence of Eliezer's partner as an organizer is just fun and quirky. I also don't know that any of the paid orgies, which Gretta also organizes, ever happened at LightHaven. I do note that Lighthaven seems desperate for money. Lighthaven's facilities manager is Ronny Hernandez, and he also moonlights as Aella's manager, and it is unclear if this just means "guy who handles her funny data science projects" or if it also means "pimp", but it probably also means "pimp". In case you haven't seen it before, Aella made this sankey chart. She has a rape kink. One of the guys on the chart is probably Eliezer Yudkowsky. Aella very aggressively advertises and recruits for her sexy lifestyle and events on Twitter.

The New York Post beat me to Nate Soares. nypost.com

twitter reactions from EA/rationalist people to the above story are. uh. they're reactions all right.

i'm gonna head off everyone who thinks this is sex negative by noting: 1) they are largely working together at a doomsday nonprofit while doing this, and 2) they recruit young idealists into this via literal harry potter fan fiction Rationalism as a whole is a religious movement. Not everyone takes marching orders. The core people who mostly live together and attend events and especially orgies in Berkeley are a sex cult, and the ones that aren't having any sex are part of the wider movement and are its recruiting funnel. Any time you get more than a few people who are deeply concerned with AI doom together, it seems like their natural form of organization looks like a cult; rationalism as a whole is less of a cult than it is a cult factory. Some of these cults have gone very badly. One of them is the Zizians.

Somewhere, some rationalist-brained person is reading this and saying I'm a "low decoupler", and that whether they run a sex cult has nothing to do with whether they're correct. This is true enough; but the sex cult is reasonable evidence that their influence isn't prevalent because they're right, and that they would have ulterior motives for continuing to preach that the world is coming to an end even if evidence showed otherwise. If you've got a sex cult and a nice compound that was given to you for free with stolen money because you preached the gospel of doom really well, how could you stop preaching it? If elaborating upon doomsday prophesies is worth money -- and it is -- you will get more of them. This all most recently came together over the weekend, and they got Dario Amodei and Sam Altman and Elon Musk and Demis Hassabis all onto the same page to say they wanted to WAIVE ANTI-TRUST LAWS so that they could "pace", or slow down, research. I feel about this roughly the same way I would feel about all the pharmaceutical companies getting together to say they wanted to waive anti-trust laws so they could form a cartel to ban medical research: these people are the enemies of humanity. Also: Much of AI research IS medical research. They are very literally proposing to stop a pretty good chunk of medical research in its tracks so that they don't have to fear the paperclipper. There are many regulations I would be in favor of, but they only ever want to form a cartel or a world government against the paperclipper. No other industry has ever framed any regulation in terms of "stop", "pause", or "slow down". Those are insane ways to consider any form of policy. Policy should prevent bad outcomes and encourage good ones, it is never about some specific speed. This industry frames its regulations in this way because it has, at its core, a millenarian death cult that worships and fears AI as the cause of the end of the world, that feels compelled to create AI, and that at least somewhat wants to destroy AI as a field. AI policy is only considered in these terms because of the vast intellectual influence of Yudkowsky specifically, and because, downstream of him, a neurotic or religious obsession with AI doom is the primary means by which people engage with these concepts. I will try to post an Abridged Rat History at some point but this is stupid long, so that will be a different thread shareable thread bsky.app