Is Aschenbrenner's 165 page paper on AI the naivety of a 25 year old?
fikisipi.substack.comIt’s pretty bizarre how SV decision makers have shifted from talking about ML to LLMs to straight up Artificial Intelligence.
I think one of the reasons is that LLMs are very good at what the execs do every day for a living and because of that, use as a way of assessing each other’s mental capacity: producing coherent-sounding speech (1).
Now that the LLM can do what any VP has dedicated their entire life to mastering, they assume that the system will be able to handle any task that they delegate, including programming, system design, project management, etc. Since the people doing this are paid less than them, surely it must be simply easier.
By this intuition, LLMs have now become intelligent, and are capable of handling anything at all. It is a must we find a way to integrate them in all of our products. And so, they’re now AI.
(1) That speech doesn’t necessarily have to carry a lot of meaning. Its main purpose is establishing the competence of the speaker.
5 years ago we felt like we had to write programs that always get the right answer. We never realized that all it had to do was apologize and people wouldn’t mind.
Eliezer Yudkowsky is up in arms about it because it can bullshit better than he can and never has to take a break. It can replace the authors of The New York Times opinion page if not the actual reporters. Can it replace the CEO?
Why not the presidency. Every four years there is a battle to fine tune the executive LLM, but a secret deep state controls the actual code running and so the LLM is modified to fake partisan fights.
p.s. only slightly tongue in cheek. Use of AI by state to manage social perception is on my bingo card. It will be very effective.
I've thought a bit about the practicality of using an LLM for a president type job. I think it could work ok with some manual override. You'd need documents with the plans and principles to follow but they'd be open source so the people could see what was going on, and if it said something silly the human leader could override but it would be public so the voters could see the LLM said do this but the humans said no that so you could keep tabs. It would make the process more open than the present system where politicians lie to get elected and then do other stuff behind closed doors.
After being initially dazzled, I've drifted firmly into the camp of "this hype cycle is nonsense". I get why VCs/Startups/investors/legislators want (or even need) a hype cycle in a post-ZIRP economy, in an election year, with geo instability. But there's nothing, neither data/evidence that LLM === AGI or even a path that points towards LLM → AGI.
AFAICT, LLMs are stochastic predictors of words within a large context. If you transfer (pun intended) that behavior over to humans, you would call a person like that a bullshitter, or at best a salesguy :-) A bullshitter as a person or team member is useful, but not scalable in the singularity sense: you can scale its output in terms of quantity, but not quality: the stochastic parrot may move prospects through the funnel (necessarily to a higher-IQ actual salesperson) but it will not create a patent from scratch (and probably will not close a deal).
So, we're not getting AGI. Given that the stochastic approach has hit scalability limits (there's no more data to feed), we need a new approach. Are there approaches that can bring AGI outside of LLMs? (AlphaZero?) Or is our industry just a bunch of stochastic parrots that complete every sentence with "eventually we'll have AGI and everything will be either great or destroyed" (which is exactly what an LLM would say at this point)?
From the article
>...The smartest people I have ever met—and they are the ones building this technology.
That's how you go beyond current LLMs. They'll figure something.
Reads like execs just have a new delusion about outsourcing the work of high paid workers to someone cheaper. Curious how this bubble will bust, or if they can fake it long enough to reach the singular point where the delusion becomes real.
SV?
Silicon Valley
It starts with the sentence
"You can see the future first in San Francisco"
and while the following paragraphs make a good, positive upbeat point, reading it out of context I cannot help thinking about homeless people in the streets and a pretty dark and dystopian future.
It's interesting how a place can be so radically different things at the same time.
> It starts with the sentence
Reminds me of William Gibson's opening line in "Neuromancer":
The sky above the port was the color of television, tuned to a dead channel.
Indeed, my perception of SF is pretty much Thunderdome.
"You can see the future first in San Francisco"
So then, accurate?
There are the very rich, that can afford to be upbeat and positive, living in a bubble.
And the poor, living in a dystopia.
Seems like it is already happening, and this a correct statement. San Francisco is a model for the future.
I think this is an important point that the folks gazing into their crystal balls overlook.
How is this article worthy of the front page of HN? It must be simply the click-bait nature of the title as there is nothing of substance beyond that.
It is a question. I tried to put what my opinion is on a few statements but I absolutely cannot summarize 160 pages (Business Insider did using GPT, which I find insulting and funny) nor have a 100% opinion on something that involves national security, secrets and other stuff that I don't have access to.
I thought it was extremely thought provoking. Are you overconfident in your weights perhaps?
Thank you.
The irony of these conversations is bizarre.
As the (tautological) saying goes: everyone is doing their best. My interest is whether this can be improved - perhaps at some point when AI gets closer to challenging us for cognitive supremacy we will awake from our slumber.
This article is blogspam. It's the paper that has people's attention.
The tone of Aschenbrenner's paper is odd. It's somewhere between his wet dreams and his nightmares. In fairness he seems to be aware of that but it's like someone grinning reflexively while they tell you bad news.
He's convinced and it's going to be terrible and it's going to be important and he's going to be part of it.
I guess for sure in 5 years it will be much clearer.
I'm familiar with this tone from my LessWrong lurker days. It comes from wrestling with the idea that there is no scalable way to actually stop AGI from coming about, so you just have to grit your teeth and hope for the best.
I mean, the numbers for the money in the paper in terms of investment mean that spending is on track for what he’s talking about. 4 to 7 more years is a lot but it’s not entirely outside the realm of possibility.
My understanding of the models and linear algebra combined with my experience of their performance improvements make me think he’s likely right. We have some people wanting to pooh-pooh these machines. They will point to current limitations of the models and weak spots as though the underlying mode of the design rather than the current implementation is the limiting factor. They act as though real intelligence is always right.
I have been wrong before but I am pretty convinced. Everyone’s arguments against it happening is that it hasn’t yet happened.
No one knows what’s on the other side of AI being a better AI researcher than humans.
Agreed with you that (a) AGI is a real, transformative, and possibly calamitous event, and (b) with the amount of money getting pumped into it we will almost certainly get there.
However, I have a personal hobby horse here, which is reminding people that if we shut the flow of investor money off using the right economic policies [1], we can almost certainly just ... stop the research outright, before it becomes an issue.
[1]: [url-redacted]
True but my view on that is that governments don't want anyone else to be the first with AI AGI... and therefore they will strive to be the first. Kinda like how a big motivation for the atomic bomb was the threat of Germany developing the bomb.
And the amount of capital means that these companies could pay a lot more for labour than they are paying now for the sorts of labour we would need to deprive them of. It's an interesting idea though.
Is it a form of apocalyptic glee? It’s quite common from some Trotskyist movements to rapture believers to conspiracy theorists… In a way it recenter the world around one single explanation/core and give a sense of, if not agency at all, cognitive control over that complexity and unknowns
I think you're right, the foundation is certainty about a huge, world-shattering change.
It's an addictive thought pattern, because it feeds the ego, provides a sense of purpose, allows escape from mundane problems, and is simple to sustain (keep believing in the Thing!)
He does address that point with a quote:
> I remember the spring of 1941 to this day. I realized then that a nuclear bomb was not only possible — it was inevitable. Sooner or later these ideas could not be peculiar to us. Everybody would think about them before long, and some country would put them into action.... And there was nobody to talk to about it, I had many sleepless nights. But I did realize how very very serious it could be. And I had then to start taking sleeping pills. It was the only remedy, I’ve never stopped since then. It’s 28 years, and I don’t think I’ve missed a single night in all those 28 years.
-- James Chadwick (corrected Chadwich -> Chadwick)
That said, I'm not sure the parallel between AGI and nuclear weapons is really that strong. Nuclear weapons are a game-changer on their own in that once you have the warhead and some kind of a delivery mechanism then you can affect events. AGIs are different in that they will manipulate and organise only information and knowledge, not physical matter directly.
Aschenbrenner sort-of-considers this point, for example
> Improved sensor networks and analysis could locate even the quietest current nuclear submarines (similarly for mobile missile launchers). Millions or billions of mouse-sized situational awareness 130 autonomous drones, with advances in stealth, could infiltrate behind enemy lines and then surreptitiously locate, sabotage, and decapitate the adversary’s nuclear forces.
...but he doesn't really consider the wider issues of, y'know, capturing the additional information that locating those nuclear submarines would require, or the technological jumps in actually creating and manufacturing those mouse-sized drones.
Perhaps the AGI is going to solve all those practical and logistical issues, but that really does remain to be seen.
Was expecting a little more analysis of or at least information about the content of the paper itself.
These posts by Zvi might be helpful (as in condensing the information, no analysis though):
https://thezvi.wordpress.com/2024/06/07/quotes-from-leopold-...
https://thezvi.wordpress.com/2024/06/10/on-dwarkshs-podcast-...
Sabine has a summary and opinion: https://www.youtube.com/watch?v=xm1B3Y3ypoE
I updated the post with a a link to counter-argument from Sabine Hossenfelder, the arguments from Zvi and three points from my side.
Here is a summary of Aschenbrenner's paper:
https://forum.effectivealtruism.org/posts/zmRTWsYZ4ifQKrX26/...
I love how the author of this article makes no intent to give even a start on why he thinks the author of the paper is naive or even describe what the paper is about and give an abstract.
Few times seen a blog post touching on so many subjects, without actually saying anything. Is there an argument, point, story, poem, news, analysis or something being made here?
> touching on so many subjects, without actually saying anything
This is a characteristic of modern rationalist/EA discourse: It’s more about opining on a subject in a heavily hedged manner that only dances around the topic. Making points too directly opens one up to being wrong, and therefore is avoided. The goal is more to wax philosophically about the subject to flex your understanding while touching multiple sides of the argument to demonstrate your breadth of knowledge, not to actually make a point.
I updated the post with a a link to counter-argument from Sabine Hossenfelder, the arguments from Zvi and three points from my side.
WHich is a proof this blog post was written by a smart human.
These guys have been waiting for their god a long time, finally they see the eclipse and believe it is the end of times
""every time a person tries to artificially make something similar using AI, people feel some interest and decide to fund research into it, but often a so called AI winter comes — and there are two AI ‘winters already (1974–1980 and 1987–2000)""
In hindsight, the winters don't look as long as they seemed living through them.
IF we want to say the last winter ended in 2000, that was 24 years ago now.
It does seem like right now, if there isn't any breaking news for a couple weeks, people think a new Winter is starting.
My question after reading the paper is: if he believe with conviction everything he wrote, why did he recently become an investor instead of either working in policy or working on the tech itself?
If the tech is going to transform humanity into a new economic paradigm and give the country controlling it a decisive advantage to be the sole superpower, being an investor is either futile or meaningless. Even doing nothing and just enjoy the ride seems to be more rational.
Doesn't he say right at the start that basically nobody is pricing this stuff in right? Becoming an investor makes a ton of sense if you earnestly believe that.
My point is that it is meaningless to make a lot of money for his scenario
Do you think all previous wealth will simply be 'reset' in some way, come the AI rapture?
Yeah, that seems like a weird assumption to make.
I guess one could make the argument that wealth will be useless in a post-scarcity scenario, so why not spend for today. But, and this might just be me speaking, I sleep a lot better at night now having some nonzero amount of money invested and going to work for me one way or another.
> why did he recently become an investor instead of either working in policy or working on the tech itself?
Even he needs a nest egg.
There's a Sabine Hossenfelder youtube just out that is more interesting than the linked article about it. "Is the Intelligence-Explosion Near? A Reality Check." https://youtu.be/xm1B3Y3ypoE
So if we supposedly already have passed the point where LLMs are at the level of high schoolers, how comes that almost no tasks or jobs that high-schoolers are capable of doing have been replaced by LLMs?
Neither will more advanced work be replaced by 2027. End of story.
I'm not on board the hype train either, but I don't think that's the right way to look at it.
High schoolers are already not widely employed for knowledge-based work. It's almost all physical labor. The few knowledge jobs among my peers in high school were tutoring and call centers, both of which are arguably replaceable with LLMs today.
But "replaced" isn't really the right frame either. In my very physically-constrained industry I can't point to any specific job titles that have been replaced by "AI" but I do see a meaningful volume of labor being taken on by ML-dependent technologies like robotics and AR, which leads to increased productivity overall which leads to bottom line results that the users are seeking.
So to me it seems reasonable that we will see this with more advanced work. The better the AI tools get the more work they will be able to assist with, including the making of better AI tools.
I did not choose this framing myself, I was responding to the paper which claimed that these jobs will be fully replaced by 2027.
And regarding your other point:
> both of which are arguably replaceable with LLMs today
Arguably, but they are not. That‘s my point.
The bigger question is whether the paper reflects the views of insiders at OpenAI.
Scott worked at OpenAI Safety and he likes it: https://scottaaronson.blog/?p=8047
But is the "-ed" in worked a problem?
An awful lot of complexity hidden behind the word 'unhobbling' here, for example page 33. Not to say we can't overcome those challenges, but they're by no means tweaks in the margins.
We must thank the author for the content-free article, so each of us can freely ramble on tangents, as if we are looking at an abstract art piece.
My beef is with the Dwarkesh Patel podcast. While he has some very good interviews (Carl Shulman, or even Zuckerberg) he seems to have a lot of rambling-on conversstion by very young employees of AI startups (openAI). I don't get value from those because they don't really say anything other than patting each other's back about how awesome they are for having been hired by their companies. I think he should focus on people with actual contribution to the science who have meaningful things to say
100% agreed. His earlier conversations before the recent AI boom were pretty great, going deep into a wide range of topics. Since then, it's taken a sharp nose-dive into AGI navel gazing. He's clearly drank too much of the kool-aid.
Aschenbrenner et al are clearly smart, but they have no real world experience to base their (entirely biased) predictions on. It's all very on-brand for the EA and EA-adjacent folks, who spend way too much time on the internet and not enough time out in the real world.
I agree, but it is the nature of the content cycle. They can’t all be winners. I very heavily curate my listening, and bounce off when it’s obvious the guest is not contributing meaningfully to my understanding of the topic.
I've been enjoying the Machine Learning Street Talk podcast, they talk to real practioners on real topics.
I love how he's got the smartest person tick on his y-axis for an "AI researcher" [in the paper]. )) These people are narcissists of the first caliber. Can we graph that?
Asked the smartest person I know and got a prediction of AGI in 1 to 2 centuries.
This is basically a self-assessed IQ test.
Did you make sure you agreed on what was meant by "AGI"?
I ask because ever since InstructGPT I've been noticing that people not only don't agree on any letter of that initialism, but also sometimes mean by it something not present in any letter.
So for me, even InstructGPT counts as a general purpose AI; for OpenAI, it's not what they mean by AGI (they require it to be economically significant); and it definitely isn't superhuman (which some people need, I'd call that ASI not merely AGI); and we still can't agree on what consciousness is let alone what it would take for an AI to have it, but some people include that in their definition of AGI and I never have.
I don't think we will agree on a definition of AGI, nor should we. To me this is settled by freedom of religion being a universal human right.
This is why the AGI in X years is such an interesting perspective.
(And why I'm deeply concerned about my downvoters.)
Treating AGI as gods is another example, yes, though I'd forgotten that one. Same direction as ASI but moreso.
People have been putting up signs on lamp posts saying "Jesus is coming" all my life.
I'll just leave it at that.
No one can predict what innovations will come. So a more accurate answer would be "I don't know, but not likely soon". But it should be pretty clear by now that the current driver of AGI talk (LLMs, possibly by extension, NNs) is not going to get us there. It might be a component of some future AGI, but not the basis
Isn't Leonardo DaVinci famous for doing just that?
Famous for doing just what?
"got a prediction of AGI in 1 to 2 centuries"
Predictions at this time scale are bullshit, at least in modern technological civilization. It would have been different in the Stone Age, but we don't live in the Stone Age, when things barely changed for millennia.
For some perspective:
100 years ago, radio was barely a thing, airplanes were made of wood and canvas, Africa was thinly populated, antibiotics didn't exist, semiconductors weren't useful yet and the British Empire reached its largest extent ever.
200 years ago, railway wasn't a thing, electric telegraphy wasn't a thing, germ theory wasn't a thing, photography wasn't a thing, most parts of Europe were still feudal, Oregon and California were still Amerindian country, the US was an unimportant country on the fringe of the developed world, and official science of the day denied the very existence of meteors as space rocks, because only dumb hicks would believe in rocks falling from the sky, ya know?
Don't tell me that people back then could make accurate predictions about the technological level of 2024. Neither can your smartest person, even though I don't doubt their smarts. Making long-term predictions is about as reliable as making long-term weather forecasts. (Not climate. Weather.)
Well thank you for assuming my friend is smart, not stupid.
200 years ago you could predict that your life would be about exactly the same as your parents' life so you had a lot of shared experience which facilitated culture transfer from one generation to the next. These days our lives are very different from our parents' life so it is difficult to share a perspective on common experiences and to transfer that wisdom. This implies there is a self-limiting factor to accelerationism.
My own prediction for AGI is a millennia at least. ;-)