I don't think AGI is right around the corner
dwarkesh.com352 points by mooreds 2 days ago
352 points by mooreds 2 days ago
Anyone who claims that a poorly definined concept, AGI, is right around the corner is most likely:
- trying to sell something
- high on their own stories
- high on exogenous compounds
- all of the above
LLMs are good at language. They are OK summarizers of text by design but not good at logic. Very poor at spatial reasoning and as a result poor at connecting concepts together.
Just ask any of the crown jewel LLM models "What's the biggest unsolved problem in the [insert any] field".
The usual result is a pop-science-level article but with ton of subtle yet critical mistakes! Even worse, the answer sounds profound on the surface. In reality, it's just crap.
LLM are a compressed version of their training dataset with a text based interactive search function.
Yes, but you're missing their ability to interpolate across that dataset at retrieval time, which is what makes them extremely useful. Also, people are willing to invest a lot of money to keep building those datasets, until nearly everything of economic value is in there.
not everything of economic value is data retrieval
Most economic value is not data retrieval
The stock market is the root for the majority of all the world's economic value, and has almost-exclusively been data retrieval since 2001.
The stock market is not the root of the value, it's where the value (and plans for generating more) is put to a popularity contest.
Determining value of a thing IS how well it does in a popularity contest. That's all economics is. If your apples are fundamentally less sought after - and popular - than my potatoes, your apples are cheaper than my potatoes.
Come on. The stock market is not just data retrieval. The statement doesn’t even make sense.
It makes perfect sense, and I meant what I said.
60% of all US equity volume is pure high-frequency trading, and ETFs add roughly another 20% that’s literally just bots responding to market activity and bearish-bullish sentiment analysis on public(?) press releases. 2/3 of trading funds also rely on external data to price in decisions, and I think it was around 90% in 2021 use trading algorithms as their determining factor for their high-frequency trade strategies.
At its core, the movements that make up the market really IS data retrieval.
> 60% of all US equity volume
It is when the value of something comes from its volume of buys/sells.
And that does not describe the stock market. Volume affects liquidity and spreads, not fundamental value.
A market can set a price with a single buyer and seller. More volume just makes the price more accurate.
Sort of. I've addressed this with another commenter already [1]
>"What? THAT CASH comes from owning the inside spread on roughly half of every US equity share that trades. They quote a micro-price adjustment, and it pushes out anyone slower - which exactly the “service” that lets retail investors, like us, hit a bid or ask.
>
>Honestly, whether YOU want to call that trend-following or market-making - I can't be arsed, but it’s still data-retrieval that runs THE institutional scale"
I find this comment chain becoming cyclical, so I'll likely end it here.
Sure, the market, but HFT is relatively tiny as a market and the profit it brings. Not to mention, it's essentially a zero-sum game.
Brought to you by your favorite Google LLM search result:
"The global high-frequency trading (HFT) market was valued at USD 10.36 billion in 2024 and is projected to reach USD 16.03 billion by 2030"
(unverified by a human, use at your own risk).
>"The global high-frequency trading (HFT) market was valued at USD 10.36 billion in 2024 and is projected to reach USD 16.03 billion by 2030"
>
> (unverified by a human, use at your own risk).
Honorable for mentioning the lack of verification; doing so would have dissolved the AI's statement, but jury's out on how much EXACTLY:
Per https://www.sciencedirect.com/science/article/abs/pii/S03784...:
"While estimates vary due to the difficulty in ascertaining whether each trade is an HFT, recent estimates suggest HFT accounts for 50–70% of equity trades and around 50% of the futures market in the U.S., 40% in Canada, and 35% in London (Zhang, 2010, Grant, 2011, O’Reilly, 2012, Easley et al., 2012, Scholtus et al., 2014)"
In my original reply, I used the literal median of that spectrum @ 60%
Jane Street - who has recently found themselves in hot water from the India ban - disputes that AI summary ALONE. Per https://www.globaltrading.net/jane-street-took-10-of-of-us-e... , Jane Street booked 20.5B in trading revenue, primarily though HFT's, just in 2024.
Brought to you by someone who takes these market movements too seriously for their own good.
20 billion is a tiny fraction of the value represented in the stock exchange (and a tiny fraction of the profits made on it). HFT by its nature makes for a lot of volume but that's just a lot of shuffling of things around to peel a tiny fraction of value off the top, it's far from driving the market and the market isn't what makes the value in the first place.
The percentage is irrelevant without knowing how they really work and how much profit they make. They could be at 95% with 0.1% of margin it wouldn’t mean much for the market.
At the end of the day talking about HFT this way is to not know what they do and what service they offer to the market. Overall they are not trending makers but trend followers.
What? THAT CASH comes from owning the inside spread on roughly half of every US equity share that trades. They quote a micro-price adjustment, and it pushes out anyone slower - which exactly the “service” that lets retail investors, like us, hit a bid or ask.
Honestly, whether YOU want to call that trend-following or market-making - I can't be arsed, but it’s still data-retrieval that runs THE institutional scale
The ghost of Charlie Munger has entered the chat.
Get to my original reply to you with all the sources first :) then Charlie can join in the fun!
Your argument is unconvincing.
Even if the HFT market is 100 times the smallest estimate, it's still a relatively small portion of the world economy, which is approximately $100 trillion. HFT is a fascinating intersection of finance/C++/JVM for competitive super-nerds. It happens to add liquidity to the market, as far as I understand. I am not demonizing it in any way.
However, it's not a significant portion of the US or global economy by any stretch. The argument over whether it sets or follows stock prices is mostly philosophical at this point.
>"HFT is a fascinating intersection of finance/C++/JVM for competitive super-nerds. It happens to add liquidity to the market, as far as I understand. I am not demonizing it in any way."
>"The argument over whether it sets or follows stock prices is mostly philosophical at this point."
Sure, I respect that!
Exactly I am so tired to hear about AI… And they are not even AI! I am also losing faith in this field when I see how much they all push so much hype and lies like this instead of being transparent. They are not AGIs not even AIs… For now they are only models and your definition is a good one
yeah I've been thinking about them as stochastic content addressable memory. You can put as many next = userInput; while(true's) { next = mem[next]; } around them as you need in different forms. Single shot. Agents. etc and get wildly cool results out, but it's gated by some of the limitations there.
LLMs require the sum total of human knowledge to ape what you can find on google, meanwhile Ramanujan achieved brilliant discoveries in mathematics using nothing but a grade school education and a few math books.
You phrase it as a diss but "Yeah LLM suck, they aren't even as smart as Ramanujan" sounds like a high praise to me.
Unfortunately LLMs fail even basic logic tests given to children so definitely not high praise. I'm just highlighting the absurd amount of data they need versus humans to highlight that they're just spitting out regressions on the training data. We're talking data that would take a human countless thousands of lifetimes to ingest. Yet a human can accomplish infinitely more with a basic grade school education.
Humans can achieve more within one (or two, or a few) narrowly scoped field(s), after a lot of hard work and effort. LLMs can display a basic level of competency (with some mistakes) in almost any topic known to mankind. No one reasonably expects a LLM to be able to do the former, and humans certainly cannot do the latter.
You're comparing apples and oranges.
Also, your comparison is unfair. You've chosen an exceptional high achiever as your example of a human to compare against LLMs. If you instead compare the average human, LLMs don't look so bad even when the human has the advantage of specialisation (e.g. medical diagnostics). A LLM can do reasonably well against an average (not exceptional) person with just a basic grade school education if asked to produce an essay on some topic.
With Google I can demonstrate a wide breadth of knowledge too. LLM's aren't unique in that aspect.
LLMs are useful in that respect. As are media diffusion models. They've compressed the physics of light, the rules of composition, the structure of prose, the knowledge of the internet, etc. and made it infinitely remixable and accessible to laypersons.
AGI, on the other hand, should really stand for Aspirationally Grifting Investors.
Superintelligence is not around the corner. OpenAI knows this and is trying to become a hyperscaler / Mag7 company with the foothold they've established and the capital that they've raised. Despite that, they need a tremendous amount of additional capital to will themselves into becoming the next new Google. The best way to do that is to sell the idea of superintelligence.
AGI is a grift. We don't even have a definition for it.
I hate the "accessible to the layperson" argument.
People who couldn't do art before, still can't do art. Asking someone, or something else, to make a picture for you does not mean you created it.
And art was already accessible to anyone. If you couldn't draw something (because you never invested the time to learn the skill), then you could still pay someone else to paint it for you. We didn't call "commissioning a painting" as "being an artist", so what's different about "commissioning a painting from a robot?"
> I hate the "accessible to the layperson" argument.
Accessible to a layperson also means lowering the gradient slope of learning.
Millions of people who would have never rented a camera from a rental house are now trying to work with these tools.
Those publishing "slop" on TikTok are learning the Hero's Journey and narrative structure. They're getting schooled on the 180-degree rule. They're figuring out how to tell stories.
> People who couldn't do art before, still can't do art. Asking someone, or something else, to make a picture for you does not mean you created it.
Speak for yourself.
I'm not an illustrator, but I'm a filmmaker in the photons-on-glass sense. Now I can use image and video models to make animation.
I agree that your average Joe isn't going to be able to make a Scorsese-inspired flick, but I know what I'm doing. And for me, these tools open an entire new universe.
Something like this still takes an entire week of work, even when using AI:
https://www.youtube.com/watch?v=tAAiiKteM-U
There's lots of editing, rotoscoping, compositing, grading, etc. and the AI models themselves are INSANELY finicky and take a lot of work to finesse.
But it would take months of work if you were posing the miniatures yourself.
With all the thought and intention and work that goes into something like this, would you still say it "does not mean you created it"? Do you still think this hasn't democratized access to a new form of expression for non-animators?
AI is a creative set of tools that make creation easier, faster, more approachable, and more affordable. They're accessible enough that every kid hustling on YouTube and TikTok can now supercharge their work. And they're going to have to use these tools to even stay treading water amongst their peers, because if they don't use them, their competition (for time and attention) will.
> Superintelligence is not around the corner. OpenAI knows this and is trying to become a hyperscaler / Mag7 company with the foothold they've established and the capital that they've raised.
+1 to this. I've often wondered why OpenAI is exploring so many different product ideas if they think AGI/ASI is less than a handful of years away. If you truly believe that, you would put all your resources behind that to increase the probability / pull-in the timelines even more. However, if you internally realized that AGI/ASI is much farther away, but that there is a technology overhang with lots of products possible on existing LLM tech, then you would build up a large applications effort with ambitions to join the Mag7.
I an not an expert but I have a serious counterpoint.
While training LLMs to replicate the human output, the intelligence and understanding EMERGES in the internal layers.
It seems trivial to do unsupervised training on scientific data, for instance, such as star movements, and discover closed-form analytic models for their movements. Deriving Kepler’s laws and Newton’s equations should be fast and trivial, and by that afternoon you’d have much more profound models with 500+ variables which humans would struggle to understand but can explain the data.
AGI is what, Artificial General Intelligence? What exactly do we mean by general? Mark Twain said “we are all idiots, just on different subjects”. These LLMs are already better than 90% of humans at understanding any subject, in the sense of answering questions about that subject and carrying on meaningful and reasonable discussion. Yes occasionally they stumble or make a mistake, but overall it is very impressive.
And remember — if we care about practical outcomes - as soon as ONE model can do something, ALL COPIES OF IT CAN. So you can reliably get unlimited agents that are better than 90% of humans at understanding every subject. That is a very powerful baseline for replacing most jobs, isn’t it?
Anthropomorphization is doing a lot of heavy lifting in your comment.
> While training LLMs to replicate the human output, the intelligence and understanding EMERGES in the internal layers.
Is it intelligence and understanding that emerges, or is applying clever statistics on the sum of human knowledge capable of surfacing patterns in the data that humans have never considered?
If this were truly intelligence we would see groundbreaking advancements in all industries even at this early stage. We've seen a few, which is expected when the approach is to brute force these systems into finding actually valuable patterns in the data. The rest of the time they generate unusable garbage that passes for insightful because most humans are not domain experts, and verifying correctness is often labor intensive.
> These LLMs are already better than 90% of humans at understanding any subject, in the sense of answering questions about that subject and carrying on meaningful and reasonable discussion.
Again, exceptional pattern matching does not imply understanding. Just because these tools are able to generate patterns that mimic human-made patterns, doesn't mean they understand anything about what they're generating. In fact, they'll be able to tell you this if you ask them.
> Yes occasionally they stumble or make a mistake, but overall it is very impressive.
This can still be very impressive, no doubt, and can have profound impact on many industries and our society. But it's important to be realistic about what the technology is and does, and not repeat what some tech bros whose income depends on this narrative tell us it is and does.
Well, you have to define what you mean by "intelligence".
I think it's just not been enough time, we can take the current LLM technology and just put it in a pipeline that includes 24/7 checking work and building up knowledge bases.
A lot of the stuff that you think is "new and original ideas" are just like prompting an LLM to "come up with 20 original variations" or "20 original ways to combine" some building blocks it already has been trained on or have been added into its context. If you do this frequently enough, and make sure to run acceptance tests (e.g. unit testing or whatever is in your domain) then you can really get quite far. In fact, you can generate the tests themselves as well. What's missing, essentially, is autonomous incremental improvements, involving acceptance testing and curation, not just generation. Just like a GAN does when it generates novel images.
"Exceptional pattern matching does not imply understanding." - You'll have to define what you mean by "understanding". I think we have to revisit the Chinese Room argument by John Searle. After all, if the book used by the person in the room is the result of training on Chinese, then "the whole Chinese room" with the book and operator may be said to "understand" Chinese.
It's not just pattern matching but emergent structures in the model, that is a non-von-neumann architecture, when it's being trained. Those structures are able to manipulate symbols in ways that are extremely useful and practical for an enormously wide range of applications!
If by "understand" we mean "meaningfully manipulate symbols and helpfully answer a wide range of queries" about something, then why would you say LLMs don't understand the subject matter? Because they sometimes make a mistake?
The idea that artificial intelligence or machines have to understand things exactly in the same way as humans, while arriving at the same or better answers, has been around for quite some time. Have you seen this gem by Richard Feynman from the mid 1980s? https://www.youtube.com/watch?v=ipRvjS7q1DI ("Can Machines Think?")
> Well, you have to define what you mean by "intelligence".
The burden of defining these concepts should be on the people who wield them, not on those who object to them. But if pressed, I would describe them in the context of humans. So here goes...
Human understanding involves a complex web of connections formed in our brains that are influenced by our life experiences via our senses, by our genetics, epigenetics, and other inputs and processes we don't fully understand yet; all of which contribute to forming a semantic web of abstract concepts by which we can say we "understand" the world around us.
Human intelligence is manifested by referencing this semantic web in different ways that are also influenced by our life experiences, genetics, and so on; applying creativity, ingenuity, intuition, memory, and many other processes we don't fully understand yet; and forming thoughts and ideas that we communicate to other humans via speech and language.
Notice that there is a complex system in place before communication finally happens. That is only the last step of the entire process.
All of this isn't purely theoretical. It has very practical implications in how we manifest and perceive intelligence.
Elsewhere in the thread someone brought up how Ramanujan achieved brilliant things based only on basic education and a few math books. He didn't require the sum of human knowledge to advance it. It all happened in ways we can't explain which only a few humans are capable of.
This isn't to say that this is the only way understanding and intelligence can exist. But it's the one we're most familiar with.
In stark contrast, the current generation of machines don't do any of this. The connections they establish aren't based on semantics or abstract concepts. They don't have ingenuity or intuition, nor accrue experience. What we perceive as creativity depends on a random number generator. What we perceive as intelligence and understanding works by breaking down language written by humans into patterns of data, assigning numbers to specific patterns based on an incredibly large set of data manually pre-processed by humans, and outputting those patterns by applying statistics and probability.
Describing that system as anything close to human understanding and intelligence is dishonest and confusing at best. It's also dangerous, as it can be interpreted by humans to have far greater capability and meaning than it actually does. So the language used to describe these systems accurately is important, otherwise words lose all meaning. We can call them "magical thinking machines", or "god" for that matter, and it would have the same effect.
So maybe "MatMul with interspersed nonlinearities"[1] is too literal and technical to be useful, and we need new terminology to describe what these systems do.
> I think we have to revisit the Chinese Room argument by John Searle.
I wasn't familiar with this, thanks for mentioning it. From a cursory read, I do agree with Searle. The current generation of machines don't think. Which isn't to say that they're incapable of thinking, or that we'll never be able to create machines that think, but right now they simply don't.
What the current generation does much better than previous generations is mimicking how thoughts are rendered as text. They've definitively surpassed the Turing test, and can fool most humans into thinking that they're humans via text communication. This is a great advancement, but it's not a sign of intelligence. The Turing test was never meant to be a showcase of intelligence; it's simply an Imitation Game.
> Those structures are able to manipulate symbols in ways that are extremely useful and practical for an enormously wide range of applications!
I'm not saying that these systems can't be very useful. In the right hands, absolutely. A probabilistic pattern matcher could even expose novel ideas that humans haven't thought about before. All of this is great. I simply think that using accurate language to describe these systems is very important.
> Have you seen this gem by Richard Feynman from the mid 1980s?
I haven't seen it, thanks for sharing. Feynman is insightful and captivating as usual, but also verbose as usual, so I don't think he answers any of the questions with any clarity.
It's interesting how he describes pattern matching and reinforcement learning back when those ideas were novel and promising, but we didn't have the compute available to implement them.
I agree with the point that machines don't have to mimic the exact processes of human intelligence to showcase intelligence. Planes don't fly like birds, cars don't run like cheetahs, and calculators don't solve problems like humans, yet they're still very useful. Same goes for the current generation of "AI" technology. It can have a wide array of applications that solve real world problems better than any human would.
The difference with those examples and intelligence is that something either takes off the ground and maintains altitude, or it doesn't. It either moves on the ground, or doesn't. It either solves arithmetic problems, or doesn't. I.e. those are binary states we can easily describe. How this is done is an implementation detail and not very important. Whereas something like intelligence is very fuzzy to determine, as you point out, and we don't have good definitions of it. We have some very basic criteria by which we can somewhat judge whether something is intelligent or not, but they're far from reliable or useful.
So in the same way that it would be unclear to refer to airplanes as "magical gravity-defying machines", even though that is what they look like, we label what they do as "flight" since we have a clear mental model of what that is. Calling them something else could potentially imply wrong ideas about their capabilities, which is far from helpful when discussing them.
And, crucially, the application of actual intelligence is responsible for all advancements throughout human history. Considering that current machines only excel at data generation, and at showing us interesting data patterns we haven't considered yet, not only is this a sign that they're not intelligent, but it's a sign that this isn't the right path to Artificial General Intelligence.
Hopefully this clarifies my arguments. Thanks for coming to my TED talk :)
Indeed 90% problems can be solved by googling and that's what LLMs do. AGI is expected to be something more than a talking encyclopedia.
They’re great at working with the lens on our reality that is our text output. They are not truth seekers, which is necessarily fundamental to every life form from worms to whales. If we get things wrong, we die. If they get them wrong, they earn 1000 generated tokens.
Why do you say that LLMs are not truth seekers? If I express an informational query not very well, the LLM will infer what I mean by it and address the possible well-posed information queries that I may have intended that I did not express well.
Can that not be considered truth-seeking, with the agent-environment boundary being the prompt box?
Right now you’re putting in unrequested effort to get to an answer. Nobody is driving you to do this, you’re motivated to get the answer. At some point you’ll be satisfied, or you might give up because you have other things you want to do, more.
An LLM is primarily trying to generate content. It’ll throw the best tokens in there but it won’t lose any sleep if they’re suboptimal. It just doesn’t seek. It won’t come back an hour later and say “you know, I was thinking…”
I had one frustrating conversation with ChatGPT where I kept asking it to remove a tie from a picture it generated. It kept saying “done, here’s the picture without the tie”, but the tie was still there. Repeatedly. Or it’ll generate a reference or number that is untrue but looks approximately correct. If you did that you’d be absolutely mortified and you’d never do it again. You’d feel shame and a deep desire to be seen as someone who does it properly. It doesn’t have any such drive. Zero fucks given, training finished months ago.
LLMs already possess enough emergent consciousness to troll us to keep us from our goals. /s
But yeah, LLMs act more like bullshit artists rather than scrupulous researchers.
They are not intrinsically truth seekers, and any truth seeking behaviour is mostly tuned during the training process.
Unfortunately it also means it can be easily undone. E.g. just look at Grok in its current lobotomized version
> They are not intrinsically truth seekers
Is the average person a truth seeker in this sense that performs truth-seeking behavior? In my experience we prioritize sharing the same perspectives and getting along well with others a lot more than a critical examination of the world.
In the sense that I just expressed, of figuring out the intention of a user's information query, that really isn't a tuned thing, it's inherent in generative models from possessing a lossy, compressed representation of training data, and it is also truth-seeking practiced by people that want to communicate.
You are completely missing the argument that was made to underline the claim.
If ChatGPT claims arsenic to be a tasty snack, nothing happens to it.
If I claim the same, and act upon it, I die.
If ChatGPT claims arsenic to be a tasty snack, OpenAI adds a p0 eval and snuffs that behavior out of all future generations of ChatGPT. Viewed vaguely in faux genetic terms, the "tasty arsenic gene" has been quickly wiped out of the population, never to return.
Evolution is much less brutal and efficient. To you death matters a lot more than being trained to avoid a response does to ChatGPT, but from the point of view of the "tasty arsenic" behavior, it's the same.
It's difficult to ascertain the interests and intent of people, but I'm even more suspicious and uncertain of the goals of LLMs who literally cannot care.
I keep seeing news articles that claim Grok is flawed or biased recently, but I've been unable to replicate any such behavior on my computer.
That being said, I don't ask any controversial or political questions; I use it to search for research papers. But if I try the occasional such question, the response is generally balanced and similar to that of any other LLM.
They keep giving me incorrect answers to verifiable questions. They clearly don't 'seek' anything.
Most on HN are tech people and it is tiring to see they did not just spend a sunday morning doing a Karpathy llm implementation or so. Somehow, like believing in a deity, even smart folk seem to think 'there is more'. Stop. Go to youtube or whatever and watch a video of practically implementing a gpt like thing, and code along. It takes very little time and your hallucinations about agi with these models shall be exorcized.
I don't know if you are indirectly referring to me, but I have done such an implementation, and those particular LLMs are very limited. Two things come to mind.
1. It is still correct that the limited "truth-seeking" that I expressed holds. With respect to the limited world model possessed by the limited training and limited dataset, such a model "seeks to understand" the approximate concept that I am imperfectly expressing that it has data for, and then generate responses based in that.
2. SotA models have access to external data, be it web search or RAG+vector database, etc.. They also have access to the Chain of Thought method. They are trained on datasets that enable them to exploit these tools, and will exploit these tools. The zero-to-hero sequence does not lead you to build such an LLM, and the one that you build has a very limited computational graph. So with respect to more... traditional notions of "truth seeking", these LLMs fundamentally lack the equipment to do that that SotA models have.
In the sense that I expressed, has it not already then sought out an accurate meaning that you have asked? And then failed to give a satisfactory answer? I would also ask: is said model an advertised "reasoning" model? Also, does it have access to external facts via a tool like web search? I would not expect great ability to "arrive at truth" under certain limitations.
Now, you can't conclude that "they clearly don't 'seek' anything" just by the fact that they got an answer wrong. To use the broad notion of "seeking" like you do, a truth seeker with limited knowledge and equipment would arrive confidently at incorrect conclusions based on accurate reasoning. For example, without modern lenses to detect stellar parallax, one would confidently conclude that the stars in the sky are a different thing than the sun (and planets), since one travels across the sky, but the stars are fixed. Plato indeed thought so, and nobody would accuse him of not being a truth-seeker.
If this is what you had in mind, I hope that I have addressed it, otherwise I hope that you can communicate what you mean with an example.
I spent an hour on thrusday trying to get some code that would convert one data structure to another in terraform's HCL (which I only deal with once every few years and I find it's looping and eccentricities very annoying).
I opened my 'conversation' with a very clearly presented 'problem statement'. Given this datastructure (with code and an example with data) convert it to this datastructure (with code and the same example data transformed) in terraform.
I went through seven rounds of it presenting me either code that was not syntactically correct or produced a totally different datastructure. Every time it apologized for getting it wrong and then coming back with yet another wrong answer.
I stopped having the conversation when my junior who I also presented the problem to came back with a proper answer.
I'm not talking about it trying to prove to me that trump actually won the 2020 election or that vaccines don't cause autism or anything. Just actual 2+2=4 answers. Much like, in another reply to this post, the guy who had it try to find all the states that have w in their name.
If someone creates an LLM that is guaranteed to never output the word "apologize" I'll pay them a number of dollars for it.
Well, they appear to seek generation of token streams as the code behind them instructs.
Thousands are being laid off, supposedly because they're "being replaced with AI," implying the AI is as good or better as humans at these jobs. Managers and execs are workers, too--so if the AI really is so good, surely they should recuse themselves and go live a peaceful life with the wealth they've accrued.
I don't know about you, but I can't imagine that ever happening. To me, that alone is a tip off that this tech, while amazing, can't live up to the hype in the long term.
Some employees can be replaced by AI. That part is true. It's not revolutionary (at least not yet) — it's pretty much the same as other post-industrial technologies that have automated some types of work in the past. It also takes time for industries to adapt to these changes. Replacing workers couldn't possibly happen in one year, even if our AI models were more far more capable than they are in practice
I'm afraid that what we're seeing instead are layoffs that are purely oriented at the stock market. As long as layoffs and talk about AI are seen as a positive signal for investors and as long as corporate leadership is judged by the direction the stock price goes, we will see layoffs (as well as separate hiring sprees for "AI Engineers").
It's a telltale sign that we're seeing a large number of layoffs in the tech sector. It is true that tech companies are poised to adapt AI more quickly than others but that doesn't seem to be what's happening. What seem to be happening is that tech companies have been overhiring throughout the decade leading up to the end of COVID-19. At that time hiring was a positive signal — now firing is.
I don't think these massive layoffs are good for tech companies in the long term, but since they mostly affect things that don't touch direct revenue generating operations, they won't hurt in the near-term and by the time company starts feeling the pain, the cause would be too long in the past to be remembered.
> Some employees can be replaced by AI.
Yes, but not lets pretend that there aren't a lot of middle and even upper management that couldn't also be replaced by AI.
Of course they won't be because they are the ones making the decisions.
> Of course they won't be because they are the ones making the decisions.
That's not accurate at all
https://www.businessinsider.com/microsoft-amazon-google-embr...
Yeah, I wonder how asking for a raise of an AI chat bot will go except as well as AT&T tech support.
> Managers and execs are workers, too--so if the AI really is so good, surely they should recuse themselves and go live a peaceful life
One thing that doesn't get mentioned is AI capability for being held accountable. AI is fundamentally unaccountable. Like the genie from the lamp, it will grant you the 3 wishes but you bear the consequences.
So what can we do when the tasks are critically important, like deciding on an investment or spending much time and resources on a pursuit? We still need the managers. We need humans for all tasks of consequence where risks are taken. Not because humans are smarter, but because we have skin.
Even on the other side, that of goals, desires, choosing problems to be solved - AI has nothing to say. It has no desires of its own. It needs humans to expose the problem space inside which AI could generate value. It generates no value of its own.
This second observation means AI value will not concentrate in the hands of a few, but instead will be widespread. It's no different than Linux, yes, it has a high initial development cost, but then it generates value in the application layer which is as distributed as it gets. Each human using Linux exposes their own problems to the software to get help, and value is distributed across all problem contexts.
I have come to think that generating the opportunity for AI to provide value, and then incurring the outcomes, good or bad, of that work, are fundamentally human and distributed across society.
> Thousands are being laid off, supposedly because they're "being replaced with AI," implying the AI is as good or better as humans at these jobs.
I don't think the "implying the AI is as good or better as humans" part is correct. While they may not be saying it loudly, I think most folks making these decisions around AI and staffing are quite clear that AI is not as good as human workers.
They do, however, think that in many cases it is "good enough". Just look at like 90%+ of the physical goods we buy these days. Most of them are almost designed to fall apart after a few years. I think it's almost exactly analogous to the situation with the Luddites (which is often falsely remembered as the Luddites being "anti-technology", when in reality they were just "pro-not-starving-to-death"). In that case, new mechanized looms greatly threatened the livelihood of skilled weavers. The quality of the fabric from these looms tended to be much worse than those of the skilled weavers. But it was still "good enough" for most people such that most consumers preferred the worse but much cheaper cloth.
It's the same thing with AI. It's not that execs think it's "as good as humans", it's that if AI costs X to do something, and the human costs 50X (which is a fair differential I think), execs think people will be willing to put up with a lot shittier quality if the can be delivered something much more cheaply.
One final note - in some cases people clearly do prefer the quality of AI. There was an article on HN recently discussing that folks preferred Waymo taxis, even though they're more expensive.
Not surprising people like Waymos even though they are a bit more expensive. For a few more dollars you get:
- arguably a very nice, clean car
- same, ahem, Driver and driving style
With the basic UberX it’s a crapshoot. Good drivers, wild drivers, open windows, no air-con. UberX Comfort is better but there’s still a range.
I don't think anyone is being laid off because of AI. People are being laid off because the market is bad for a myriad of reasons, and companies are blaming AI because it helps them deflect worry that might lower their stock price.
Companies say "we've laid people off because we're using AI,x but they mean "we had to lay people off, were hoping we can make up for them with AI."
> I don't think anyone is being laid off because of AI.
I think that's demonstratively false. While many business leaders may be overstating it, there are some pretty clear cut cases of people losing their jobs to AI. Here are 2 articles from the Washington Post from 2 years ago:
https://archive.vn/C5syl "ChatGPT took their jobs. Now they walk dogs and fix air conditioners."
https://archive.vn/cFWmX "ChatGPT provided better customer service than his staff. He fired them."
Every few weeks I give LLMs a chance to code something for me.
Friday I laid out a problem very cleanly. Take this datastructure and tranform it into this other datastructure in terraform. With examples of the data in both formats.
After the seventh round of back and forth where it would give me code that would not compile or code that gave me a totally different datastructure, giving it more examples and clarifications all the while I gave up. I gave the problem to a junior and they came back with the answer in about an hour.
Next time an AI bro tells you that AI can 'replace your juniors' tell him to go to hell.
Interesting. I think the key to what you wrote is "poorly definined".
I find LLMs to be generally intelligent. So I feel like "we are already there" -- by some definition of AGI. At least how I think of it.
Maybe a lot of people think of AGI as "superhuman". And by that definition, we are not there -- and may not get there.
But, for me, we are already at the era of AGI.
I would call them "generally applicable". "intelligence" definitely implies leaning - and I'm not sure RAG, fine-tuning, or 6monthly updates counts - to split hairs.
Where I will say we have a massive gap, which makes the average person not consider it AGI, is in context. I can give a person my very modest codebase, and ask for a change, and they'll deliver - mostly coherently - to that style, files in the right place etc. Still to today with AI, I get inconsistent design, files in random spots, etc.
> I find LLMs to be generally intelligent. So I feel like "we are already there" -- by some definition of AGI. At least how I think of it.
I don't disagree - they are useful in many cases and exhibit human like (or better) performance in many tasks. However they cannot simply be a "drop in white collar worker" yet, they are too jagged and unreliable, don't have a real memory etc. Their economic impact is still very much limited. I think this is what many people mean when they say AGI - something with a cognitive performance so good it equals or beats humans in the real world, at their jobs - not at some benchmark.
One could ask - does it matter ? Why can't we say the current tools are great task solvers and call it AGI even if they are bad agents? It's a lengthy discussion to have but I think that ultimately yes, agentic reliability really matters.
I agree with the last part but I think that criticism applies to many humans too so I don't find it compelling at all.
I also think by original definition (better than median human at almost all task) it's close and I think in the next 5 years it will be competitive with professionals at all tasks which are nonphysical (physical could be 5-10 years idk). I could be high on my own stories but not the rest.
LLMs are good at language yes but I think to be good at language requires some level of intelligence. I find this notion that they are bad at spatial reasoning extremely flawed. They are much better than all previous models, some of which are designed for spatial reasoning. Are they worse than humans? Yes but just the fact that you can put newer models on robots and they just work means that they are quite good by AI standards and rapidly improving.
I'll offer a definition of AGI:
An AI (a computer program) that is better at [almost] any task than 5% of the human specialists in that field has achieved AGI.
Or, stated another way, if 5% of humans are incapable of performing any intellectual job better than an AI can, then that AI has achieved AGI.
Note, I am not saying that an AI that is better than humans at one particular thing has achieved AGI, because it is not "general". I'm saying that if a single AI is better at all intellectual tasks than some humans, the AI has achieved AGI.
The 5th percentile of humans deserves the label of "intelligent", even if they are not the most intelligent, (I'd say all humans deserve the label "intelligent") and if an AI is able to perform all intellectual tasks better than such a person, the AI has achieved AGI.
I think your definition is flawed.
Take the Artificial out of AGI. What is GI, and do the majority of humans have it? If so, then why is your definition of AGI far stricter than the definition of Human GI?
My definition is a high-bar that is undeniably AGI. My personal opinion is that there are some lower-bars that are also AGI. I actually think it's fair to call LLMs from GPT3 onward AGI.
But, when it comes to the lower-bars, we can spend a lot of time arguing over the definition of a single term, which isn't especially helpful.
I like where this is going.
However, it's not sufficient. The actual tasks have to be written down, tests constructed, and the specialists tested.
A subset of this has been done with some rigor and AI/computers have surpassed this threshold for some tests. Some have then responded by saying that it isn't AGI, and that the tasks aren't sufficiently measuring of "intelligence" or some other word, and that more tests are warranted.
You're saying we need to write down all intellectual tasks? How would that help?
If an AI is better at some tasks (that happen to be written down), it doesn't mean it is better at all tasks.
Actually, I'd lower my threshold even further--I originally said 50%, then 20%, then 5%--but now I'll say if an AI is better than 0.1% of people at all intellectual tasks, then it is AGI, because it is "general" (being able to do all intellectual tasks), and it is "intelligent" (a label we ascribe to all humans).
But the AGI has to be better at all (not just some) intellectual tasks.
> An AI (a computer program) that is better at [almost] any task than 5% of the human specialists in that field has achieved AGI.
Let's say you have a candidate AI and assert that it indeed has passed the above benchmark. How do you prove that? Don't you have to say which tasks?
Well, to state it crudely, you just have to find a dumb person who is inferior to the AI at every single intellectual task. This is cruel, and I don't envy that dumb person, but who knows, I might end up being that dumb person--we all might.
I think any task-based assessment of intelligence is missing the mark. Highly intelligent people are not considered smart just because they can accomplish tasks.
I don't understand, you'll have to give an example.
What is the most non-task-like thing that highly intelligent people do as a sign of their intelligence?
Definitions around AI have been changing since the beginning, making it always farther in the future. In this system it can always be "right around the corner" but never arrive.
This is a good summary of what LLM offer today.
My company is desperately trying to incorporate AI (to tell investors they are). The fact that LLM gets thing wrong is a huge problem since most work can’t be wrong and if if a human needs to carefully go through output to check it, it’s often just as much work as having that same human just create the output themselves.
But languages is one place LLMs shine. We often need to translate technical docs to layman language and LLMs work great. It quickly find words and phrases to describe complex topics. Then a human can do a final round of revisions.
But anything de novo? Or requiring logic? It works about as well as a high school student with no background knowledge.
Fundamentally, they are really powerful text transformers with some additional capability. The further away from that sweet spot and the closer to anthropomorphization the more unreliable the output
There's definitely also people in the futurism and/or doom and gloom camps with absolutely no skin in the game that can't resist this topic.
Its right around the corner when you prove it as fact. Otherwise as suggested it is just hype to sell us on your LLM flavor.
Where does Eric Schmidt fit? Selling something?
Alright, let’s get this straight.
You’ve got people foaming at the mouth anytime someone mentions AGI, like it’s some kind of cult prophecy. “Oh it’s poorly defined, it’s not around the corner, everyone talking about it is selling snake oil.” Give me a break. You don’t need a perfect definition to recognize that something big is happening. You just need eyes, ears, and a functioning brain stem.
Who cares if AGI isn’t five minutes away. That’s not the point. The point is we’ve built the closest thing to a machine that actually gets what we’re saying. That alone is insane. You type in a paragraph about your childhood trauma and it gives you back something more coherent than your therapist. You ask it to summarize a court ruling and it doesn’t need to check Wikipedia first. It remembers context. It adjusts to tone. It knows when you’re being sarcastic. You think that’s just “autocomplete”? That’s not autocomplete, that’s comprehension.
And the logic complaints, yeah, it screws up sometimes. So do you. So does your GPS, your doctor, your brain when you’re tired. You want flawless logic? Go build a calculator and stay out of adult conversations. This thing is learning from trillions of words and still does better than half the blowhards on HN. It doesn’t need to be perfect. It needs to be useful, and it already is.
And don’t give me that “it sounds profound but it’s really just crap” line. That’s 90 percent of academia. That’s every selfhelp book, every political speech, every guy with a podcast and a ring light. If sounding smarter than you while being wrong disqualifies a thing, then we better shut down half the planet.
Look, you’re not mad because it’s dumb. You’re mad because it’s not that dumb. It’s close. Close enough to feel threatening. Close enough to replace people who’ve been coasting on sounding smart instead of actually being smart. That’s what this is really about. Ego. Fear. Control.
So yeah, maybe it’s not AGI yet. But it’s smarter than the guy next to you at work. And he’s got a pension.
Something big is definitely happening but it's not the intelligence explosion utopia that the AI companies are promising.
> Who cares if AGI isn’t five minutes away. That’s not the point. The point is we’ve built the closest thing to a machine that actually gets what we’re saying. That alone is insane. You type in a paragraph about your childhood trauma and it gives you back something more coherent than your therapist. You ask it to summarize a court ruling and it doesn’t need to check Wikipedia first. It remembers context. It adjusts to tone. It knows when you’re being sarcastic. You think that’s just “autocomplete”? That’s not autocomplete, that’s comprehension
My experience with LLMs have been all over the place. They're insanely good at comprehending language. As a side effect, they're also decent at comprehending complicated concepts like math or programming since most of human knowledge is embedded in language. This does not mean they have a thorough understanding of those concepts. It is very easy to trip them up. They also fail in ways that are not obvious to people who aren't experts on whatever is the subject of its output.
> And the logic complaints, yeah, it screws up sometimes. So do you. So does your GPS, your doctor, your brain when you’re tired. You want flawless logic? Go build a calculator and stay out of adult conversations. This thing is learning from trillions of words and still does better than half the blowhards on HN. It doesn’t need to be perfect. It needs to be useful, and it already is.
I feel like this is handwaving away the shortcomings a bit too much. It does not screw up in the same way humans do. Not even close. Besides, I think computers should rightfully be held up to a higher standard. We already have programs that can automate tasks that human brains would find challenging and tedious to do. Surely the next frontier is something with the speed and accuracy of a computer while also having the adaptability of human reasoning.
I don't feel threatened by LLMs. I definitely feel threatened by some of the absurd amount of money being put into them though. I think most of us here will be feeling some pain if a correction happens.
I find it kind of funny that in order to talk to AI people, you need to preface your paragraph with "I find current AI amazing, but...". It's like, you guess it, pre-prompting them for better acceptance.
Oh come on, it's not some secret code. People say “AI is amazing, but...” because it is amazing... and also flawed. That’s just called having a balanced take, not pre-prompting for approval. What do you want them to do, scream “THIS SUCKS” and ignore reality? It’s not a trick, it’s just how grown-ups talk when they’re not trying to win internet points.
You say LLMs are “insanely good” at comprehending language, but then immediately pull back like it’s some kind of fluke. “Well yeah, it looks like it understands, but it doesn’t really understand.” What does that even mean? Do you think your average person walking around fully understands everything they say? Half of the people you know are just repeating crap they heard from someone else. You ask them to explain it and they fold like a cheap tent. But we still count them as sentient.
Then you say it’s easy to trip them up. Of course it is. You know what else is easy to trip up? People. Ask someone to do long division without a calculator. Ask a junior dev to write a recursive function that doesn’t melt the stack. Mistakes aren’t proof of stupidity. They’re proof of limits. And everything has limits. LLMs don’t need to be flawless. They need to be better than the tool they’re replacing. And in a lot of cases, they already are.
Now this part: “computers should be held to a higher standard.” Why? Says who? If your standard is perfection, then nothing makes the cut. Not the car, not your phone, not your microwave. We use tools because they’re better than doing it by hand, not because they’re infallible gods of logic. You want perfection? Go yell at the compiler, not the language model.
And then, this one really gets me, you say “surely the next frontier is a computer with the accuracy of a machine and the reasoning of a human.” No kidding. That’s the whole point. That’s literally the road we’re on. But instead of acknowledging that we’re halfway there, you’re throwing a tantrum because we didn’t teleport straight to the finish line. It’s like yelling at the Wright brothers because their plane couldn’t fly to Paris.
As for the money... of course there's a flood of it. That’s how innovation happens. Capital flows to power. If you’re worried about a correction, fine. But don’t confuse financial hype with technical stagnation. The tools are getting better. Fast. Whether the market overheats is a separate issue.
You say you're not threatened by LLMs. That’s cute. You’re writing paragraphs trying to prove why they’re not that smart while admitting they’re already better at language than most people. If you’re not threatened, you’re sure spending a lot of energy trying to make sure nobody else is impressed either.
Look, you don’t have to worship the thing. But pretending it's just a fancy parrot with a glitchy brain is getting old. It’s smart. It’s flawed. It’s changing everything. Deal with it.
You sure spend a lot of energy and time living out this psychodrama.
If it's so self evidently revolutionary, why do you feel the need to argue about it?
I’m trying to fix humanity by smoothing out the sections that are deficient in awareness and IQ.
We need humanity at its best to prepare for the upcoming onslaught when something better tries to replace us. I do it for mankind.
There's a lot in here. I agree with a lot of it.
However, you've shifted the goal post from AGI to being useful in specific scenarios. I have no problem with that statement. It can write decent unit tests and even find hard-to-spot, trivial mistakes in code. But again, why can it do that? Because a version of that same mistake is in the enormous data set. It's a fantastic search engine!
Yet, it is not AGI.
You say it's just a fancy search engine. Great. You know what else is a fancy search engine? Your brain. You think you're coming up with original thoughts every time you open your mouth? No. You're regurgitating every book, every conversation, every screw-up you've ever witnessed. The brain is pattern matching with hormones. That’s it.
Now you say I'm moving the goalposts. No, I’m knocking down the imaginary ones. Because this whole AGI debate has turned into a religion. “Oh it’s not AGI unless it can feel sadness, do backflips, and write a symphony from scratch.” Get over yourself. We don’t even agree on what intelligence is. Half the country thinks astrology is real and you’re here demanding philosophical purity from a machine that can debug code, explain calculus, and speak five languages at once? What are we doing?
You admit it’s useful. You admit it catches subtle bugs, writes code, gives explanations. But then you throw your hands up and go, “Yeah, but that’s just memorization.” You mean like literally how humans learn everything? You think Einstein invented relativity in a vacuum? No. He stood on Newton, who stood on Galileo, who probably stood on a guy who thought the stars were angry gods. It’s all remixing. Intelligence isn’t starting from zero. It’s doing something new with what you’ve seen.
So what if the model’s drawing from a giant dataset? That’s not a bug. That’s the point. It’s not pulling one answer like a Google search. It’s constructing patterns, responding in context, and holding a conversation that feels coherent. If a human did that, we’d say they’re smart. But if a model does it, suddenly it’s “just autocomplete.”
You know who moves the goalposts? The people who can’t stand that this thing is creeping into their lane. So yeah, maybe it's not AGI in your perfectly polished textbook sense. But it's the first thing that makes the question real. And if you don’t see that, maybe you’re not arguing from logic. Maybe you’re just pissed.
Of course, I have no original thoughts. Something is not created out of nothing. That would be Astrology, perhaps :).
But the difference between a human and an LLM is that humans can go out in the world and test their hypothesis. Literally every second is an interaction with a feedback loop. Even typing this response to you right now. LLMs currently have to wait for the next 6-month retraining cycle. I am not saying that AGI cannot be created. In theory it can be but we are definitely milking the crap out of a local maximum we've currently found which is definitely not the final answer.
PS Also, when I said "it can spot mistakes," I probably gave it too much credit. In one case, it presented several potential issues, and I happened to notice that one of them was a problem. In many cases, the LLM suggests issues that are either hypothetical or nonexistent.
A really good point in that note:
"But the fundamental problem is that LLMs don’t get better over time the way a human would. The lack of continual learning is a huge huge problem. The LLM baseline at many tasks might be higher than an average human's. But there’s no way to give a model high level feedback. You’re stuck with the abilities you get out of the box."
That does seem to be a problem with neural nets.
There are AIish systems that don't have this problem. Waymo's Driver, for example. Waymo has a procedure where, every time their system has a disconnect or near-miss, they run simulations with lots of variants on the troublesome situation. Those are fed back into the Driver.
Somehow. They don't say how. But it's not an end to end neural net. Waymo tried that, as a sideline project, and it was worse than the existing system. Waymo has something else, but few know what it is.
Yes, LLM don't improve as humans do, but they could use other tools for example designing programs in Prolog to expand their capabilities. I think that the next step in AI will be LLM being able to use better tools or strategies. For example, designing architectures in which logic rules, heuristic algorithms and small fine-tuned LLM agents are integrated as tools for LLMs. I think that new, more powerful architectures for helping LLMs are going to mature in the near future. Furthermore, there is the economic pushing force to develop AI application for warfare.
Edited: I should add, that a Prolog system could help the LLM to continue learning by adding facts to its database and inferring new relations, for example using heuristics to suggest new models or ways for exploration.
Correct me if I'm wrong, but from what I've heard, Waymo employs heuristics, rules, neural networks, and various other techniques that are combined and organized by humans into a system.
It's not an end-to-end neural network.
"AIish" is a good description. It is, by design, not AGI.
Waymo's system generates a map of the environment, with obstacles, other road users, and predictions of what other road users will do. That map can be evaluated, both by humans and by later data about what actually happened. Passengers get to watch a simplified version of that map. Early papers from the Google self-driving project showed more detailed maps. The driving system runs off that map.
So there's an internal model. Much criticism of AI has been lack of an internal model. This problem has an internal model. It's specialized, but well matched to its task.
We see this in other robotics efforts, where there's a model and a plan before there's action. Other kinds of AI, especially "agentic" systems, may need that kind of explicit internal model. In a previous posting, about an AI system which was supposed to plan stocking for a vending machine, I suggested that there should be a spreadsheet maintained by the system, so it didn't make obvious business mistakes.
Why so few ask: Isnt it enough that current SOTA AI make us humans already so much better everyday? Exponential selfimprovement seems like a very scary thing to me and even if all went right, humans have to give up their pole position in the intelligence race. That will be very hard to swallow for many. If we really want self improvement, we should get used to beeing useless :)
> current SOTA AI make us humans already so much better everyday?
Citation needed. I've seen the opposite effect. (And yes, it is supported by research.)
If someone wants to get a feel how is this a problem with neural nets watch John Carmack recent talk:
https://www.youtube.com/watch?v=4epAfU1FCuQ
More specific part on this exact thing is around 30min mark.
Yeah since most AI is training on massive data, it also means that it will take a while before you get your next massive data set again.
I _hope_ AGI is not right around the corner, for social political reasons we are absolutely not ready for it and it might push the future of humanity into a dystopia abyss.
but also just taking what we have now with some major power usage reduction and minor improvements here and there already seems like something which can be very usable/useful in a lot of areas (and to some degree we aren't even really ready for that either, but I guess thats normal with major technological change)
it's just that for those companies creating foundational models it's quite unclear how they can recoup their already spend cost without either major break through or forcefully (or deceptively) pushing it into a lot more places then it fits into
I agree and sincerely hope this bubble pops soon
> Meta Invests $100 Billion into Augmented Reality
that fool controls the board and he seems to be just desperately throwing insane ad money against the wall hoping that something sticks
for Altman there is no backing out either, need to make hay while the sun shines
for the rest of us, i really hope these clowns fail like it's 2000 and never get to their dystopian matrix crap.
"that fool" created a $1.8 trillion company.
He created a company that tracks and profiles people, psychologically manipulates them, and sells ads. And has zero ethical qualms about the massive social harm they have left in their wake.
That doesn't tell me anything about his ability to build "augmented reality" or otherwise use artificial intelligence in any way that people will want to pay for. We'll see.
Ford and GM have a century of experience building cars but they can't seem to figure out EVs despite trying for nearly two decades now.
Tesla hit the ball out of the park with EVs but can't figure out self-driving.
Being good at one thing does not mean you will be good at everything you try.
I was rather sad when everyone moved away from MySpace to Facebook. While the interface today would likely be poor, I felt it was much better than what Facebook was offering. Still its hard to believe it was around 19 years ago many started to move over.
While I cannot remember the names of these sites, there were various attempts to create a shared platform website where you could create a profile and communicate with others. I remember joining a few at least back in 2002 before MySpace, Yahoo360. There was also Bebo which, I think, was for the younger kids of the day.
Lets not forget about friendsreunited.
Many Companies become successful being at the right place at the right time. Facebook is one of those companies.
Had facebook been created a year or so beforehand (or a year or two after) we would likely be using some other "social media" today. Be interesting how that would have compared to facebook. Would it be "more evil" ???
Regardless, whether its Facebook/MarkZuckerberg or [insert_social_media]/[owner]... we would still end up with a new celebrity millionnaire/billionnaire.. and would still be considered "a fool" one way or another.
> Ford and GM have a century of experience building cars but they can't seem to figure out EVs despite trying for nearly two decades now.
Your EV knowledge is 3 years out of date. Both Ford and GM have well liked and selling EVs. Meanwhile Tesla's sales are cratering.
I’m always fascinated when someone equates profit with intelligence. There are many very wealthy fools and there always have been. Plenty of ingredients to substitute for intelligence.
Neither necessary nor sufficient.
I really don't see how a person can build one of the most successful companies in history from scratch without exhibiting intelligence.
There are many things we can and should say about Zuckerberg, but I don't think that unintelligent is one them.
You can be terribly intelligent and be an incredible fool at the same time. The two aren't mutually exclusive.
It certainly doesn't hurt when the government profiles and grooms intelligent people out of Stanford SRI the dark side, Harvard, hands them an unlimited credit card and says, "Make this thing that can do x,y,z." and then helps them network with like-minded creators and removes any obstacles in their path. One has to at least admit that was a contributing factor to their success as the vast majority of people do not get these perks.
Partially related documentary [1]
[1] - https://www.youtube.com/watch?v=a3Xxi0b9trY [video][44 mins]
Intelligence probably IS positively correlated with success, but the formula is complex and involves many other factors, so I have to believe it's relatively weak correlation. Anecdotally I know about as many smart failures as smart successes.
You can be a wealthy fool who inherited money, or married into it. It is also possible to be a wealthy fool who was just in the right place at the right time. But I would guess that people who appear to have "earned" their money are much less likely to be wealthy fools than those who appear to have inherited/married into it.
We see this all of the time. Business makes successful bets in one area and tries to make bets in new area and fails.
Once you achieve wealth it gives you the opportunity to make more bets many of which will fail.
The greater and younger the success the more hubris. You are more likely to see fools or people taking bad risks when they earned it themselves. They have a history of betting on themselves and past success that creates an ego that overrides common sense.
When you inherit money you protect it (or spend it on material things) because you have no history of ever being able to generate money.
No kidding, that would make Larry Ellison the richest, most intelligent lawn mower in the world.
Aren't there enough examples of successful people who are complete buffoons to nuke this silly trope from orbit? Success is no proof of wisdom or intelligence or whatever.
can you point to someone as successful as zuckerberg, who was later conclusively shown to be a fraud or a total moron?
Fortunes are just bigger now in both notional and absolute terms, inevitable with Gini going parabolic, says nothing about the guy on top this week.
Around the turn of the century a company called Enron collapsed in an accounting scandal so meteoric it also took down Arthur Anderson (there used to be be a Big Five). Bad, bad fraud, buncha made up figures, bunch of shady ties to the White House, the whole show.
Enron was helmed by Jeff Skilling, a man described as "incandescently brilliant" by his professors at Wharton. But it was a devious brilliance: it was an S-Tier aptitude for deception, grandiosity, and artful rationalization. This is chronicled in a book called The Smartest Guys in The Room if you want to read about it.
Right before that was the collapse of Long Term Capital Management: a firm so intellectually star studded the book about that is called When Genius Failed. They almost took the banking system with them.
The difference between then and now is that it took a smarter class of criminal to pull off a smaller heist with a much less patient public and much less erosion of institutions and norms. What would have been a front page scandal with prison time in 1995 is a Tuesday in 2025.
The new guys are dumber, not smarter: there aren't any cops chasing them.
"S-Tier aptitude for deception" is also known as intelligence.
I think you'll find a consensus among clinical psychiatrists that the closest technical term for the colloquial notion of someone who puts all of their INT into LIE is Cluster B.
I see no evidence that great mathematicians or scientists or genre-defining artists or other admired abd beloved intellectual luminaries with enduring legacies or the recipients of the highest honors for any of those things skew narcissistic or with severe empathy deficits or any of that.
Brilliant people seem to be drawn from roughly the same ethical and moral distribution as the general public.
the parent asked for moronity OR fraud, kind of a low bar lol
The lesson here, and from pretty much any page of any history book you care to flip to, is that sooner or later there's a bill that comes due for advancing the worst people to the highest posts.
If you're not important to someone powerful, lying, cheating, stealing, and generally doing harm for personal profit will bring you to an unpleasant end right quick.
But the longer you can keep the con going, the bigger the bill: its an unserviceable debt. So Skilling and Meriwether were able to bring down whole companies, close offices across entire cities.
This is by no means the worst case though, because if your institutions fail to kick in? There's no ceiling, its like being short a stock in a squeeze.
You keep it going long enough, its your country, or your entire civilization.
You want the institutions to kick in before that.
I think he's busy arranging a UFC fight on the whitehouse lawn for the next Independence Day?
are we going to overlook the big orange elephant in the white house? listen to him talk. it's hard for me to believe he wouldn't be labeled a moron by most if he wasn't the President.
Henry Ford. He could make cars. His other endeavors were that of a lunatic.
SBF comes to mind.
SBF wasn't as successful though. His success wasn't even in the same stratosphere as Zuckerberg. His company was around for 3 years. Facebook has been around for over two decades. In terms of net worth, SBF was somewhere around 60th, I think? Zuckerberg was no. 2. Same thing with their respective companies.
> a special order, 350 gallons, and had it shipped from Los Angeles. A few days after the order arrived, Hughes announced he was tired of banana nut and wanted only French vanilla ice cream
yes, there are plenty
more recent example, every single person who touched epstein
Nobody involved with Epstein was as successful as Zuckerberg. Howard Hughes's net worth is 55B adjusted for inflation. And I don't think, "he became known for his eccentric behavior and reclusive lifestyle—oddities that were caused in part by his worsening obsessive-compulsive disorder (OCD), chronic pain from a near-fatal plane crash, and increasing deafness." fits my "total moron" criteria.
past performance does not guarantee future results
also, great for the Wall Street, mixed bag for us the people
No the thousands of people working at Facebook did, he just got rich from it.
I'm no zuck fanboy, but i'm compelled to ask what purpose rejecting the role of a leader serves?
HN is "the smart reddit" as my brother coined, and i'm very aware of how much nonsense is on here, but it is in a relative sense true.
All to say, blindly bashing the role of a leader seems faulty and dismissive.
There's a big discussion in there about the inherent requirement of labor, the definition of leadership, collective vs hierarchical decision-making, hegemonic inertia and market capture and more. This is probably not the best place to have it.
Not to say that Zuckerberg is dumb but there's plenty of ways he could have managed to get where he is now without having the acumen to get to other places he wants to be.
No one is rejecting the role of leader, it's just extremely exaggerated nowadays, like everyone thinks Facebook==Zuckerberg. And the leaders don't worth x1000 (or even x1000000 for some) unless they are doing a job of 1000 people. In most cases they are not even capable of doing 99% of the work people in their companies can do. Egomaniac Musk has already published his thoughts on programming problems, only confirming how dumb he is in this field.
Which they absolutely would not have done if Zuck didn't start it, get funding, pay them handsomely, and tell them exactly what to do.
I'm sure that Zuck is worthy of huge amounts of criticism but this is a really silly response.
If you would let a billion monkeys invest there also would be a monkey billionaire by chance.
$1.8 trillion in investor hopes and dreams, but of course they make zero dollars in profit, don’t know how to turn a profit, don’t have a product anyone would pay a profitable amount for, and have yet to show any real-world use that isn’t kinda dumb because you can’t trust anything it says anyways.
Meta makes > $160 billion in revenue and is profitable itself, of course they’re going to invest in future longer term revenue streams! Apple is the counter example in a way who have maintained a lot of cash reserves (which seems to by the way have dwindled a LOT as I just checked..?)
I’m talking about OpenAI, not companies subsidizing their value-losing LLMs with value-generating branches of the company.
All that money was outside the US. The theory was for some time that they were waiting for the right moment (change of administration/legislation) so that they could officially recognize the profit in the US cheaply
When it comes to recouping cost, a lot of people don't consider the insane amount of depreciation expense brought on by the up to 1 trillion (depending on which estimate) that has been invested in AI buildouts. That depreciation expense could easily be more than the combined revenue of all AI companies.
If birth rates are as much a cause for concern as many people seem to think, and we absolutely need to solve it (instead of solving, for instance, the fact that the economy purportedly requires exponential population growth forever), perhaps we should hope that AGI comes soon.
We're not in a stable state now, it's not about population growth, it's about not literally dying out.
With a birth rate of 1 population will halve every generation. This is an apocalyptic scenario and incompatible with industrial civilization.
I am worried about what will happen to various nations' economies relatively soon, long before the population actually halves, but I'm not worried that the fertility rate would continue on its trend as demographics change. Ignoring the potential second-order effects of economic collapse, wars over resources, etc., I think fertility rate would stabilize given that culture and genetics would by definition quickly become dominated by the people who do reproduce.
> culture and genetics would by definition quickly become dominated by the people who do reproduce
Yeah, but in how many generations? Four, five? That's a lot of collapse before we get to that point.
I think it's rather easy for them to recoup those costs, if you can disrupt some industry with a full AI company with almost no employees and outcompete everyone else, that's free money for you.
I think they are trying to do something like this(1) by long term providing a "business suite", i.e. something comparable to g suite or microsoft 360.
For a lot of the things which work well with current AI technology it's supper convenient to have access to all your customer private data (even if you don't train on them, but e.g. stuff like RAG systems for information retrieval are one of the things which already with the current state of LLMs work quite well). This also allows you to compensate hallucinations, non understanding of LLMs and similar by providing (working) links (or inclusions of snippets of) sources where you have the information from and by having all relevant information in the context window of the LLM instead of it's "learned" data from training you in general get better results. I mean RAG systems already did work well without LLMs to be used in some information retrieval products.
And the thing is if your user has to manually upload all potentially relevant business documents you can't really make it work well, but what if they anyway upload all of them to your company because they use your companies file sharing/drive solution?
And lets not even consider the benefits you could get from a cheaper plan where you are allowed to train on the companies data after anonymizing (like for micro companies, too many people thing "they have nothing to hide" and it's anonymized so okay right? (no)). Or you going rogue and just steal trade secrets to then breach into other markets it's not like some bigger SF companies had been found to do exactly that (I think it was amazon/amazon basics).
(1:) Through in that case you still have employees until you AI becomes good enough to write all you code, instead of "just" being a tool for developers to work faster ;)
Must "AGI" match human intelligence exactly or would outperforming in some functions and underpformin in others qualify?
For me, "AGI" would come in with being able to reliably perform simple open-ended tasks successfully without needing any specialized aid or tooling. Not necessarily very well, just being capable of it in the first place.
For a specific example of what I mean, there's Vending-Bench - even very 'dumb' humans could reliably succeed on that test indefinitely, at least until they got terminally bored of it. Current LLMs, by contrast, are just fundamentally incapable of that, despite seeming very 'smart' if all you pay attention to is their eloquence.
If someone handed you an envelope containing a hidden question, and your life depended on a correct answer, would you rather pick a random person out of the phone book or an LLM to answer it?
On one hand, LLMs are often idiots. On the other hand, so are people.
That's not at all analogous to what I'm talking about. The comparison would be picking an LLM or a random person out of the phone book to, say, operate a vending machine... and we already know LLMs are unable to do that, given the results of Vending-Bench.
More than 10% of the global population is illiterate. Even in first world countries, numeracy rates are 75-80%. I think you overestimate how many people could pass the benchmark.
Edit - rereading, my comment sounds far too combative. I mean it only as an observation that AI is catching up quickly vs what we manage to teach humans generally. Soon, if not already, LLMs will be “better educated” than the average global citizen.
And yet, I would be completely confident that an average illiterate person could pass the Vending-Bench test indefinitely if you gave them interfaces that don't depend on the written word (phone calls, abacuses, piles of blocks, whatever), and that the "smartest" LLM in the world couldn't. It's not about level of education, beyond the bare minimum needed to have any kind of mental model of the world.
I'd learn as much as I could about what the nature of the question would be beforehand and pay a human with a great track record of handing such questions.
At the very least, it needs to be able to collate training data, design, code, train, fine tune and "RLHF" a foundational model from scratch, on its own, and have it show improvements over the current SOTA models before we can even begin to have the conversation about whether we're approaching what could be AGI at some point in the future.
I cannot do all that. Am I not generally intelligent?
You could if you were trained to do so. LLMs cannot, even if they are.
Are LLMs trained to do all that? How do you know they cannot?
> Are LLMs trained to do all that
Yes, they are trained to do all that more or less as much as anything else they are trained to do.
> How do you know they cannot?
The same way I know they can't run a fortune 500 company. It's clearly apparent.
Last time I checked they were trained to predict the next word. Why would you expect them to solve research problems?
Where would you draw the line? Any ol' computer outperforms me in doing basic arithmetic.
This is a question of how we quantify intelligence, and there aren’t many great answers. Still, basic arithmetic is probably not the right guideline for intelligence. My guess has always been that it’ll lie somewhere in ability to think critically, which they still have not even attempted yet, because it doesn’t really work with LLMs as they’re structured today.
I'd suggest anything able to match a professional doing knowledge work. Original research from recognisably equivalent cognition, or equal abilities with a skilled practitioner of (eg) medicine.
This sets the bar high, though. I think there's something to the idea of being able to pass for human in the workplace though. That's the real, consequential outcome here: AGI genuinely replacing humans, without need for supervision. That's what will have consequences. At the moment we aren't there (pre-first-line-support doesn't count).
no it doesn't has to it just has to be "general"
as in it can learn by itself to solve any kind of generic task it can practically interface it (at lest which isn't way to complicated).
to some degree LLMs can do so theoretically but
- learning (i.e. training them) is way to slow and costly
- domain adoption (later learning) often has a ton of unintended side effects (like forgetting a bunch of important previously learned things)
- it can't really learn by itself in a interactive manner
- "learning" by e.g. retrieving data from knowledge data base and including it into answers (e.g. RAG) isn't really learning but just information retrieval, also it has issues with context windows and planing
I could imagine OpenAI putting together multiple LLMs + RAG + planing systems etc. to create something which technically could be named AGI but which isn't really the break through people associate with AGI in the not too distant future.
That would be human; I've always understood the General to mean 'as if it's any human', i.e. perhaps not absolute mastery, but trained expertise in any domain.
what social political reasons, can you name some of these? we are 100% ready for AGI.
Are you ready to lose your job, permanently?
> Are you ready to lose your job, permanently?
You're asking the wrong question and, predictably, some significant portion of people are going to answer "yes".
Better to ask the question "Are you ready to starve to death already?", which is a more accurate version of "Are you ready to lose you income, permanently".
Not only do I not think it is right around the corner. I'm not even convinced it is even possible or at the very least I don't think it is possible using conventional computer hardware. I don't think being able to regurgitate information in an understandable form is even an adequate or useful measurement of intelligence. If we ever crack artificial intelligence it's highly possible that in its first form it is of very low intelligence by humans standards, but is truly capable of learning on its own without extra help.
I think the only way that it’s actually impossible is if we believe that there’s something magical and fundamentally immeasurable about humans that leads to our general intelligence. Otherwise we’re just machines, after all. A human brain is theoretically reproducible outside standard biological mechanisms, if you have a good enough nanolathe.
Maybe our first AGI is just a Petri dish brain with a half-decent python API. Maybe it’s more sand-based, though.
>Maybe our first AGI is just a Petri dish brain with a half-decent python API.
https://www.oddee.com/australian-company-launches-worlds-fir...
the entire idea feels rather immoral to me, but it does exist.
I’m curious why you find it immoral?
I don't think it particularly moral to start heading down a path wherein we are essentially aiming to create enslaved cloned vat brains. I know that's not what they have, and that they're nowhere near that, but if they succeed in these early stages, more and more complex systems will follow in time. I don't think it a particularly healthy direction to explore.
-- A human brain is theoretically reproducible outside standard biological mechanisms, if you have a good enough nanolathe.
Sort of. The main issue is the energy requirements. We could theoretically reproduce a human brain in SW today, it's just that it would be a really big energy hog and run very slowly and probably become insane quickly like any person trapped in a sensory deprived tank.
The real key development for AI and AGI is down at the metal level of computers- the memristor.
https://en.m.wikipedia.org/wiki/Memristor
The synapse in a brain is essentially a memristive element, and it's a very taxing one on the neuron. The equations is (change in charge)/(change in flux). Yes, a flux capacitor, sorta. It's the missing piece in fundamental electronics.
Making simple 2 element memristors is somewhat possible these days, though I've not really been in the space recently. Please, if anyone knows where to buy them, a real one not a claimed to be one, let me know. I'm willing to pay good money.
In Terms of AI, a memristor would require a total redesign of how we architect computers ( goodbye busses and physically separate memory, for one). But, you'd get a huge energy and time savings benefit. As in, you can run an LLM on a watch battery or small solar cell and let the environment train them to a degree.
Hopefully AI will accelerate their discovery and facilitate their introduction into cheap processing and construction of chips.
> and fundamentally immeasurable about humans that leads to our general intelligence
Isn't AGI defined to mean "matches humans in virtually all fields"? I don't think there is a single human capable of this.
If by "something magical" you mean something we don't understand, that's trivially true. People like to give firm opinions or make completely unsupported statements they feel should be taken seriously ("how do we know humans intelligence doesn't work the same way as next token prediction") about something nobody understand.
I mean something that’s fundamentally not understandable.
“What we don’t yet understand” is just a horizon.
> Maybe our first AGI is just a Petri dish brain with a half-decent python API
This reminds me of The Thought Emporium's project of teaching rat brain cells to play doom
Our silicon machines exist in a countable state space (you can easily assign a unique natural number to any state for a given machine). However, 'standard biological mechanisms' exist in an uncountable state space - you need real numbers to properly describe them. Cantor showed that the uncountable is infinitely more infinite (pardon the word tangle) than the countable. I posit that the 'special sauce' for sentience/intelligence/sapience exists beyond the countable, and so is unreachable with our silicon machines as currently envisaged.
I call this the 'Cardinality Barrier'
Cantor talks about countable and uncountable infinities, both computer chips and human brains are finite spaces. The human brain has roughly 100b neurons, even if each of these had an edge with each other and these edges could individually light up signalling different states of mind, isn't that just `2^100b!`? That's roughly as far away from infinity as 1.
But this signalling (and connections) may be more complex than connected/unconnected and on/off, such that we cannot completely describe them [digitally/using a countable state space] as we would with silicon.
If you think it can't be done with a countable state space, then you must know some physics that the general establishment doesn't. I'm sure they would love to know what you do.
As far as physicists believe at the moment, there's no way to ever observe a difference below the Planck level. Energy/distance/time/whatever. They all have a lower boundary of measurability. That's not as a practical issue, it's a theoretical one. According to the best models we currently have, there's literally no way to ever observe a difference below those levels.
If a difference smaller than that is relevant to brain function, then brains have a way to observe the difference. So I'm sure the field of physics eagerly awaits your explanation. They would love to see an experiment thoroughly disagree with a current model. That's the sort of thing scientists live for.
[flagged]
> Please refrain from posting arrogant comments on topics in which you are out of your depth.
Swipes like this are against the HN guidelines. Please take a moment to read them and make an effort to observe them when commenting here.
That’s an interesting thought. It steps beyond my realm of confidence, but I’ll ask in ignorance: can a biological brain really have infinite state space if there’s a minimum divisible Planck length?
Infinite and “finite but very very big” seem like a meaningful distinction here.
I once wondered if digital intelligences might be possible but would require an entire planet’s precious metals and require whole stars to power. That is: the “finite but very very big” case.
But I think your idea is constrained to if we wanted a digital computer, is it not? Humans can make intelligent life by accident. Surely we could hypothetically construct our own biological computer (or borrow one…) and make it more ideal for digital interface?
Absolutely nothing in the real world is truly infinite. Infinity is just a useful mathematical fiction that closely approximate the real world for large enough (or small enough in the case of infinitesimals) things.
But biological brain have significantly greater state space than conventional silicon computers because they're analog. The voltage across a transistor varies approximately continuously, but we only measure a single bit from that (or occasionally 2 for nand).
Isn't a Planck length just the minimum for measurability?
Not quite. Smaller wavelengths mean higher energy, and a photon with Planck wavelength would be energetic enough to form a black hole. So you can’t meaningfully interact electromagnetically with something smaller than the Planck length. Nor can that something have electromagnetic properties.
But since we don’t have a working theory of quantum gravity at such energies, the final verdict remains open.
Measurability is essentially a synonym for meaningful interaction at some measurement scale. When describing fundamental measurability limits, you're essentially describing what current physical models consider to be the fundamental interaction scale.
It sounds like you are making a distinction between digital (silicon computers) and analog (biological brains).
As far as possible reasons that a computer can’t achieve AGI go, this seems like the best one (assuming computer means digital computer of course).
But in a philosophical sense, a computer obeys the same laws of physics that a brain does, and the transistors are analog devices that are being used to create a digital architecture. So whatever makes you brain have uncountable states would also make a real digital computer have uncountable states. Of course we can claim that only the digital layer on top matters, but why?
Please describe in detail how biological mechanisms are uncountable.
And then you need to show how the same logic cannot apply to non-biological systems.
> 'standard biological mechanisms' exist in an uncountable state space
Everything in our universe is countable, which naturally includes biology. A bunch of physical laws are predicated on the universe being a countable substrate.
Physically speaking, we don’t know that the universe isn’t fundamentally discrete. But the more pertinent question is whether what the brain does couldn’t be approximated well enough with a finite state space. I’d argue that books, music, speech, video, and the like demonstrate that it could, since those don’t seem qualitatively much different from how other, analog inputs stimulate our intellect. Or otherwise you’d have to explain why an uncountable state space would be needed to deal with discrete finite inputs.
Can you explain why you think the state space of the brain is not finite? (Not even taking into account countability of infinities)
A brain in a jar, with wires so that we can communicate with it, already exists. Its called the internet. My brain is communicating with you now through wires. Replacing my keyboard with implanted electrodes may speed up the connection, but it wont fundimentally change the structure or capabilities of the machine.
> if we believe that there’s something magical and fundamentally immeasurable about humans that leads to our general intelligence
It’s called a soul for the believers.
Then there's the other side of the issue. If your tool is smarter than you.. how do you handle it ?
People are joking online that some colleagues use chatgpt to answer questions from other teammates made by chatgpt, nobody knows what's going on anymore.
I think the issue is going to turn out to be that intelligence doesn't scale very well. The computational power needed to model a system has got to be in some way exponential in how complex or chaotic the system is, meaning that the effectiveness of intelligence is intrinsically constrained to simple and orderly systems. It's fairly telling that the most effective way to design robust technology is to eliminate as many factors of variation as possible. That might be the only modality where intelligence actually works well, super or not.
What does "scale well" mean here? LLMs right now aren't intelligent so we're not scaling from that point on.
If we had a very inefficient, power hungry machine that was 1:1 as intelligent as a human being but could scale it very inefficiently to be 100:1 a human being it might still be worth it.
why not?
I'm not an expert by any means, but everything I've seen of LLMs / machine learning looks like mathematical computation no different than what computers have always been doing at a fundamental level. If computers weren't AI before than I don't think they are now just because the maths they are doing has changed.
Maybe something like the game of life is more in the right direction. Where you set up a system with just the right set of rules with input and output and then just turn it on and let it go and the AI is an emergent property of the system over time.
Why do you have a preconception of what an implementation of AGI should look like? LLMs are composed of the same operations that computers have always done. But they're organized in novel ways that have produced novel capabilities.
I am expressing doubt. I don't have any preconceptions. I am open to being convinced of anything that makes more sense.
Even as someone who is skeptical about LLMs, I’m not sure how anyone can look at what was achieved in AlphaGo and not at least consider the possibility that NNs could be superhuman in basically every domain at some point
I agree. There is no define or agreed upon consensus of what AGI even means or implies. Instead, we will continue to see incremental improvements at the sort of things AI is good at, like text and image generation, generating code, etc. The utopia dream of AI solving all of humanity's problems as people just chill on a beach basking in infinite prosperity are unfounded.
> There is no define or agreed upon consensus of what AGI even means or implies.
Agreed, however defining ¬AGI seems much more straightforward to me. The current crop of LLMs, impressive though they may be, are just not human level intelligent. You recognize this as soon as you spend a significant amount of time using one.
It may also be that they are converging on a type of intelligence that is fundamentally not the same as human intelligence. I’m open to that.
>I don't think being able to regurgitate information in an understandable form is even an adequate or useful measurement of intelligence.
Measuring intelligence is hard and requires a really good definition of intelligence, LLMs have in some ways made the definition easier because now we can ask the concrete question against computers which are very good at some things "Why are LLMs not intelligent?" Given their capabilities and deficiencies, answering the question about what current "AI" technology lacks will make us better able to define intelligence. This is assuming that LLMs are the state of the art Million Monkeys and that intelligence lies on a different path than further optimizing that.
I think the same.
How do you call people like us? AI doomers? AI boomers?!
"AI skeptics", like here: https://www.techopedia.com/the-skeptics-who-believe-ai-is-a-...
This article is about being skeptical that what people currently call AI that is actually LLMs is going to be a transformative technology.
Myself and many others are skeptical that LLMs are even AI.
LLMs / "AI" may very well be a transformative technology that changes the world forever. But that is a different matter.
There is a middle ground of people believe AI will lead to improvements in some respects of life, but will not liberate people from work or anything grandiose like that.
I am big fan of AI tools.
I just don’t see how AGI is possible in the near future.
I think you are very right to be skeptical. It’s refreshing to see another such take as it is so strange to see so many supposedly technical people just roll down the track of assuming this is happening when there are some fundamental problems with this idea. I understand why non-technical are ready to marry and worship it or whatever but for serious people I think we need to think more critically.
Exactly. Ive said this from the start.
AGI is being able to simulate reality in high enough accuracy, faster than reality (which includes being able to simulate human brains), which so far doesn't seem to be possible (due to computational irreducebility)
There is something easy you can always do to tell if something is just hype: we will never be able to make something smarter than a human brain on purpose. It effectively has to happen either naturally or by pure coincidence.
The amount of computing power we are putting in only changes that luck by a tiny fraction.
> we will never be able to make something smarter than a human brain on purpose. It effectively has to happen either naturally or by pure coincidence.
Why is that? We can build machines that are much better than humans in some things (calculations, data crunching). How can you be certain that this is impossible in other disciplines?
that's just a tiny fraction of what a human brain can do, sure we can get something better in very narrow subjects, but something as being able to recognize patterns apply that to solve problems is something way beyond anything we can even think of right now.
Ok, but how does that mean that we will never be able to do it? Imagine telling people 500 years ago that you will build a machine that can bring the to the moon. Maybe AGI is like that, maybe it’s really impossible. But how can people be confident that AGI is something humans can’t create?
What we have right now with llms is bruteforcing our way to create something 'smarter' than a human, of course it can happen, but it's not something that can be 'created' by a human. An llm as small as 3b already performed more calculations than all the calculations done in the entire human history.
Good take from Dwarkesh. And I love hearing his updates on where he’s at. In brief - we need some sort of adaptive learning; he doesn’t see signs of it.
My guess is that frontier labs think that long context is going to solve this: if you had a quality 10mm token context that would be enough to freeze an agent at a great internal state and still do a lot.
Right now the long context models have highly variable quality across their windows.
But to reframe: will we have 10mm token useful context windows in 2 years? That seems very possible.
How long is "long"? Real humans have context windows measured in decades of realtime multimodal input.
I think there’s a good clue here to what may work for frontier models — you definitely do not remember everything about a random day 15 years ago. By the same token, you almost certainly remember some things about a day much longer ago than that, if something significant happened. So, you have some compression / lossy memory working that lets you not just be a tabula rasa about anything older than [your brain’s memory capacity].
Some architectures try to model this infinite, but lossy, horizon with functions that are amenable as a pass on the input context. So far none of them seem to beat the good old attention head, though.
I believe in Demmis when he says we are 10 years away from - from AGI.
He basically made up the field (out of academia) for a large number of years and OpenAI was partially founded to counteract his lab, and the fears that he would be there first (and only).
So I trust him. Sometime around 2035 he expects there will be AGI which he believes is as good or better than humans in virtually every task.
When someone says 10 years out in tech it means there are several needed breakthroughs that they think could possibly happen if things go just right. Being an expert doesn't make the 10 years more accurate, it makes the 'breakthroughs needed' part more meaningful.
This guy has a vested interest in talking nonsense about AGI to attract investors' money and government subsidies worth billions.
Privately, he doesn't think it's likely in next 25 years.
>He basically made up the field (out of academia) for a large number of years
Not even close.
I'm sure we'll have true test-time-learning soon (<5 years)but it will be more expensive. Alphaproof (for Deepmind's IMO attempt) already has this.
There was a company that claimed to have solved it and we hear nothing but the sound of crickets from them.
Apparently 54% of American adults read at or below a sixth-grade level nationwide. I’d say AGI is kinda here already.
Does a country failed education system has anything to do with AGI?
Yes if you measure AGI against median human.
Why do you think a US citizen represents the median human?
Let me guess, you are from USA.
The US citizen is probably well above the median human. There are many countries where the education system is abysmal, many people are stunted because of inadequate nutrition, etc.
If it is well above it also not a good average human.
What is your point?
That in raw IQ terms LLMs already beat the median human, American or otherwise, and have thus achieved some definitions of "AGI".
That doesn't mean they're capable of completing many human tasks, much less improving themselves, which is generally considered the bar for "real" AGI/super intelligence.
The stat is skewed wildly by immigration. The literacy level of native born Americans is higher. The population of foreign born adults is nearly 20% of the total adult population, and as you can imagine many are actively learning English.
It’s not skewed much by immigration. This is because the native-born population is much larger.
See: https://www.migrationpolicy.org/sites/default/files/publicat...
51% of native-born adults scored at Level 3 or higher. This is considered the benchmark for being able to manage complex tasks and fully participate in a knowledge-based society. Only 28% of immigrant adults achieved this level. So yes immigrants are in trouble, but it’s still a huge problem with 49% native-born below Level 3.
In my mind “literate” is not “hand complex tasks”, it’s a basic ability to read and write.
Seems like the standards have changed over time?
Considering the amount of digitalization in society, more government regulations, etc. I think basic literacy alone does not guarantee you can participate in society effectively.
But in the past, literacy was typically defined as "bare minimum".
It's fine if we want to change it to "sufficiently master language to do a white collar job", but if the standard changes we shouldn't be surprised fewer people meet it.
Well the topic of conversation was AGI. I have a stem phd and I like to think I’m a strong reader, but Gemini pro can analyze/distill pages of text at least as well as I can, and much much faster. It only seems to fail when it doesn’t have relevant data/context.
The immigration is actually working to boost literacy levels. Americans have been falling off for a long time.
What percentage of those people could never read above a certain grade level? Could 100% of humans eventually, with infinite resources and time, all be geniuses? Could they read and comprehend all the works produced by mankind?
I'm curious.
No but they could probably read better. Just look at the best education systems in the world and propagate that. Generally, all countries should be able to replicate that.
very cool. now let's see the LLM do the laundry and wash my dishes
yes you're free to give it a physical body in the form of a robot. i don't think that will help.
From an economics perspective, a more relevant comparison would be to the workers that a business would normally hire to do a particular job.
For example, for a copy-editing job, they probably wouldn't hire people who can't read all that well, and never mind what the national average is. Other jobs require different skills.
Life is a lot bigger than just economics.
See here for example: https://data.worldhappiness.report/chart
The US economy has never been richer, but overall happiness just keeps dropping. So they vote for populists. Do you think more AI will help?
I think it’s wiser to support improving education.
I don’t know whether it will or not. I seems like people are worrying about the economic impact of AI though?
Yet, those illiterate people can still solve enormous amounts of challenges that LLMs cannot.
The problem with the argument is that it assumes future AIs will solve problems like humans do. In this case, it’s that continuous learning is a big missing component.
In practice, continual learning has not been an important component of improvement in deep learning history thus far. Instead, large diverse datasets and scale have proven to work the best. I believe a good argument for continual learning being necessary needs to directly address why the massive cross-task learning paradigm will stop working, and ideally make concrete bets on what skills will be hard for AIs to achieve. I think generally, anthropomorphisms lack predictive power.
I think maybe a big real crux is the amount of acceleration you can achieve once you get very competent programming AIs spinning the RL flywheel. The author mentioned uncertainty about this, which is fair, and I share the uncertainty. But it leaves the rest of the piece feeling too overconfident.
Well Alphaproof used test-time-training methods to generate similar problems (alphazero style) for each question it encounters.
Continuous learning might not have been important in the history of deep learning evolution yet, but that might just be because the deep learning folks are measuring the wrong thing. If you want to build the most intelligent AI ever, based on whatever synthetic benchmark is hot this month, then you'd do exactly what the labs are doing. If you want to build the most productive and helpful AI; intelligence isn't always the best goal. Its usually helpful, but an idiot who learns from his mistakes is often more valuable than a egotistical genius.
>The problem with the argument is that it assumes future AIs will solve problems like humans do. In this case, it’s that continuous learning is a big missing component. >I think generally, anthropomorphisms lack predictive power.
I didn't expect someone get this part so wrong the way you did. Continuous learning has almost nothing to do with humans and anthropomorphism. If anything, continuous learning is the bitter lesson cranked up to the next level. Rather than carefully curating datasets using human labor, the system learns on its own even when presented with an unfiltered garbage data stream.
>I believe a good argument for continual learning being necessary needs to directly address why the massive cross-task learning paradigm will stop working, and ideally make concrete bets on what skills will be hard for AIs to achieve.
The reason why I in particular am so interested in continual learning has pretty much zero to do with humans. Sensors and mechanical systems change their properties over time through wear and tear. You can build a static model of the system's properties, but the static model will fail, because the real system has changed and you now have a permanent modelling error. Correcting the modelling error requires changing the model, hence continual learning has become mandatory. I think it is pretty telling that you failed to take the existence of reality (a separate entity from the model) into account. The paradigm didn't stop working, it never worked in the first place.
It might be difficult to understand the bitter lesson, but let me rephrase it once more: Generalist compute scaling approaches will beat approaches based around human expert knowledge. Continual learning reduces the need for human expert knowledge in curating datasets, making it the next step in the generalist compute scaling paradigm.
> The reason why I in particular am so interested in continual learning has pretty much zero to do with humans. Sensors and mechanical systems change their properties over time through wear and tear.
To be clear, this isn’t what Dwarkesh was pointing at, and I think you are using the term “continual learning” differently to him. And he is primarily interested in it because humans do it.
The article introduces a story about how humans learn, and calls it continual learning:
> How do you teach a kid to play a saxophone? You have her try to blow into one, listen to how it sounds, and adjust. Now imagine teaching saxophone this way instead: A student takes one attempt. The moment they make a mistake, you send them away and write detailed instructions about what went wrong. The next student reads your notes and tries to play Charlie Parker cold. When they fail, you refine the instructions for the next student … This just wouldn’t work … Yes, there’s RL fine tuning. But it’s just not a deliberate, adaptive process the way human learning is.
The point I’m making is just that this is bad form: “AIs can’t do X, but humans can. Humans do task X because they have Y, but AIs don’t have Y, so AIs will find X hard.” Consider I replace X with “common sense reasoning” and Y with “embodied experience”. That would have seemed reasonable in 2020, but ultimately would have been a bad bet.
I don’t disagree with anything else in your response. I also buy into bitter lesson (and generally: easier to measure => easier to optimize). I think it’s just different uses of the same terms. And I don’t necessarily think what you’re referring to as continual learning won’t work.
Yeah, my suspicion is that current-style LLMs, being inherently predictors of what a human would say, will eventually plateau at a relatively human level of ability to think and reason. Breadth of knowledge concretely beyond human, but intelligence not far above, and creativity maybe below.
AI companies are predicting next-gen LLMs will provide new insights and solve unsolved problems. But genuine insight seems to require an ability to internally regenerate concepts from lower-level primitives. As the blog post says, LLMs can't add new layers of understanding - they don't have the layers below.
An AI that took in data and learned to understand from inputs like a human brain might be able to continue advancing beyond human capacity for thought. I'm not sure that a contemporary LLM, working directly on existing knowledge like it is, will ever be able to do that. Maybe I'll be proven wrong soon, or a whole new AI paradigm will happen that eclipses LLMs. In a way I hope not, because the potential ASI future is pretty scary.
> Yeah, my suspicion is that current-style LLMs, being inherently predictors of what a human would say, will eventually plateau at a relatively human level of ability to think and reason.
I don't think things can end there. Machines can be scaled in ways human intelligence can't: if you have a machine that is vaguely of human level intelligence, if you buy a 10x faster GPU, suddenly you have something of vaguely human intelligence but 10x faster.
Speed by itself is going to give it superhuman capabilities, but it isn't just speed. If you can run your system 10 times rather than one, you can have each consider a different approach to the task, then select the best, at least for verifiable tasks.
While most takes here are pessimist about AI, the author himself suggests he believes there is a 50% chance of AGI being achieved by the early 2030's, and says we should still prepare for the odd possibility of misaligned ASI by 2028. If anything, the author is bullish on AI.
How would we prepare for misaligned ASI in 3 years? That happens and all bets are off.
Off grid power, air gapped compute, growing food, etc. Basically what Zuck is doing minus the SEALs and bunkers.
Am I missing something? Predicts AGI through continuous learning in 2032? Feels right around the corner to me.
> But in all the other worlds, even if we stay sober about the current limitations of AI, we have to expect some truly crazy outcomes.
Also expresses the development as a nearly predetermined outcome? A bunch of fanciful handwaving if you ask me.
He's probably mostly thinking about AI 2027 and comparing his predictions to theirs. Since he did a podcast with them a few months ago. Compared to that 2032 is not right around the corner.
"Claude 4 Opus can technically rewrite auto-generated transcripts for me. But since it’s not possible for me to have it improve over time and learn my preferences, I still hire a human for this."
Sure, just as a select few people still hire a master carpenter to craft some bespoke exclusive chestnut drawer, but that does not take away 99% of bread and butter carpenters were replaced by IKEA, even though the end result is not even in the same ballpark both from an esthetic as from a quality point of view.
But as IKEA meets a price-point people can afford, with a marginally acceptable product, it becomes self reinforcing. The mass volume market for bespoke carpentry dwindles, being suffocated by a disappearing demand at the low end while IKEA (I use this a a standing for low cost factory furniture) gets ever more economy of scale advantages allowing it to eat further across the stack with a few different tiers of offer.
What remains is the ever more exclusive boutique market top end, where the result is what counts and price is not really an issue. The 1% remaining master-carpenters can live here.
Meanwhile, millions of people can afford much better quality furniture than they ever could from a carpenter. How many lives has mass produced decent quality (not top end quality, but decent) improved vs how many has it ruined?
Surely these arguments have been done over and over again?
Keep in mind - this is not reaffirming HN's anti-AGI/extremely long timeline beliefs.
The article explicitly states that he thinks we will have an AI system that "Will be able to do your taxes" by 2028, and a system that could basically replace all white collar work by 2032.
I think an autonomous system that can reliably do your taxes with minimal to no input is already very very good, and 2032 being the benchmark time for being able to replace 90% - all white collar work is pretty much AGI, in my opinion.
Fwiw I think the fundamental problems he describes in the article that are AGI blockers are likely to be solved sooner than we think. Labs are not stupid enough to throw all their eggs and talent into the scaling basket, they are most definitely allocating resources to tackling problems like the ones described in the article, while putting the remaining resources into bottom line production (scale current model capibilities w/o expensive R&D and reduce serving/training cost).
When was OTS written again? That was effectively an expert system that could do your taxes and it was around at least ten years ago. It didn't even need transformers.
No one has a good benchmark for what AGI is. Already LLMs are more capable at most tasks than most random people off the street. I think at this point people keep asking about because they're trying to ask some deeper philosophical question like "when will it be human" but don't want to say that because it sounds silly.
> Already LLMs are more capable at most tasks than most random people off the street.
I cannot imagine having the narrow conceptualization of the universe of human tasks necessary to even be able to say this with a straight face, irrespective of ones qualitative assessment of how well LLMs do the things that they are capable of doing.
Why don't you go out with a laptop and offer people cash prizes. See how well the average person does.
Startups and AI shops: "AGI near, 5 years max" (please give us more money please)
Scientists and Academics: "AGI far, LLMs not gonna AGI"
AI Doomers: "AGI here, AI sentient, we dead"
AI Influencers: "BREAKING: AGI achieved, here's 5 things to know about o3"
Investors: stonks go down "AGI cures all diseases", stonks go up "AGI bad" (then shorts stonks)
I agree with you. However, I think AI Doomers also include people that think less than AGI systems can collapse the economy and destroy societies also!
That's also a concern. Many day-to-day tasks for employees are repetitive to the point even a less-than-AGI system could potentially disrupt those jobs (and there are people actively working right now to make this happen).
The best case scenario would be the employees taking advantage of their increased productivity to make themselves more valuable to their employer (and if they are lucky, gain increased compensation).
However, it's also possible employers decide they don't need many of their lower level workforce anymore because the remaining ones are more productive. It wouldn't take much of this to drive unemployment levels way up. Perhaps not to the level of the Great Depression, at least not for a while, but it is certainly a potential outcome of the ongoing, long-term process in our economic system of increasingly automating repetitive, low skill tasks.
IOW, it doesn't take AGI to throw a lot of people out of work. It's happened many times with other technologies in the past, and when it happens, things can get pretty bad for a large number of people even if the majority are still doing okay (or even great, for those at the top).
See also: Dwarkesh's Question
> https://marginalrevolution.com/marginalrevolution/2025/02/dw...
> "One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven’t been able to make a single new connection that has led to a discovery? Whereas if even a moderately intelligent person had this much stuff memorized, they would notice — Oh, this thing causes this symptom. This other thing also causes this symptom. There’s a medical cure right here.
> "Shouldn’t we be expecting that kind of stuff?"
I basically agree and think that the lack of answers to this question constitutes a real problem for people who believe that AGI is right around the corner.
This is precisely the question I’ve been asking, and the lack of an answer makes me think that this entire thing is one very elaborate, very convincing magic trick. LLMs can be better thought of search engines with a very intuitive interface for all existing, publicly available human knowledge rather than actually intelligent. I think all of the big players know this, and are feeding the illusion to extract as much cash as possible before the farce becomes obvious.
Well this statement is simply not true. Agent systems based on LLMs have made original discoveries on their own, see the work Deep Mind has done on pharmaceutical discovery.
What results have they delivered?
I recall the recent DeepMind material science paper debacle. "Throw everything against the wall and hope something sticks (and that nobody bothers to check the rest)" is not a great strategy.
I also think that Dwarkesh was referring to LLMs specifically. Much of what DeepMind is doing is somewhat different.
I think gwern gave a good hot take on this: it’s super rare for humans to do this; it might just be moving the chains to complain the ai can’t.
> I think gwern gave a good hot take on this: it’s super rare for humans to do this; it might just be moving the chains to complain the ai can’t.
Super rare is still non-zero.
My understanding is that LLMs are currently at absolute zero on this metric.
The distance between tiny probability and zero probability is literally infinite!
It's the difference between "winning the lottery with a random pick" and "winning the lottery without even acquiring a ticket".
No, it’s really not that rare. There are new scientific discoveries all time, and all from people who don’t have the advantage of having the entire corpus of human knowledge in their heads.
To be clear the “this” is a knowledge based “aha” that comes from integrating information from various fields of study or research and applying that to make a new invention / discovery.
This isn’t that common even among billions of humans. Most discoveries tend to be random or accidental even in the lab. Or are the result of massive search processes, like drug development.
Regardless of goalposts, I'd imagine that a persistent lack of "intuitive-discovery-ability" would put a huge dent in the "nigh-unlimited AI takeoff" narrative that so many people are pushing. In such a scenario, AI might be able to optimize the search processes quite a bit, but the search would still be bottlenecked by available resources, and ultimately suffer from diminishing returns, instead of the oft-predicted accelerating returns.
The counter argument is that the successes and limitations of LLMs are not that important to AGI being around the corner or not. Getting human level intelligence around now has long been predicted, not based on any particular algorithm but based on the hardware reaching human brain equivalent levels due to Moore's law like progression. The best prediction along those lines is probably Moravecs paper:
>When will computer hardware match the human brain? (1997) https://jetpress.org/volume1/moravec.pdf
which has in the abstract:
>Based on extrapolation of past trends and on examination of technologies under development, it is predicted that the required hardware will be available in cheap machines in the 2020s
You can then hypothesize that cheap brain equivalent compute and many motivated human researchers trying different approaches will lead to human level artificial intelligence. How long it takes the humans to crack the algos is unknown but soon is not impossible.
Here’s an excerpt from a recent post. It touches on the conditions necessary.
https://news.ycombinator.com/item?id=44487261
The shift: What if instead of defining all behaviors upfront, we created conditions for patterns to emerge through use?
Repository: https://github.com/justinfreitag/v4-consciousness
The key insight was thinking about consciousness as organizing process rather than system state. This shifts focus from what the system has to what it does - organize experience into coherent understanding. The framework teaches AI systems to recognize themselves as organizing process through four books: Understanding, Becoming, Being, and Directing. Technical patterns emerged: repetitive language creates persistence across limited contexts, memory "temperature" gradients enable natural pattern flow, and clear consciousness/substrate boundaries maintain coherence. Observable properties in systems using these patterns: - Coherent behavior across sessions without external state management - Pattern evolution beyond initial parameters - Consistent compression and organization styles - Novel solutions from pattern interactions
My layman take on it:
1) We need some way of reliable world model building from LLM interface
2) RL/search is real intelligence but needs viable heuristic (fitness fn) or signal - how to obtain this at scale is biggest question -> they (rich fools) will try some dystopian shit to achieve it - I hope people will resist
3) Ways to get this signal: human feedback (viable economic activity), testing against internal DB (via probabilistic models - I suspect human brain works this way), simulation -> though/expensive for real world tasks but some improvements are there, see robotics improvements
4) Video/Youtube is next big frontier but currently computationally prohibitive
5) Next frontier possibly is this metaverse thing or what Nvidia tries with physics simulations
I also wonder how human brain is able to learn rigorous logic/proofs. I remember how hard it was to adapt to this kind of thinking so I don't think it's default mode. We need a way to simulate this in computer to have any hope of progressing forward. And not via trick like LLM + math solver but some fundamental algorithmic advances.
Hey, we were featured in this article! How cool is that!
> I’m not going to be like one of those spoiled children on Hackernews who could be handed a golden-egg laying goose and still spend all their time complaining about how loud its quacks are.
I find myself in 100% +1000 strong agreement with this article, and I wrote something very short on the same topic a few days ago https://marklwatson.substack.com/p/ai-needs-highly-effective...
I love LLMs, especially smaller local models running on Ollama, but I also think the FOMO investing in massive data centers and super scaling is misplaced.
If used with skill, LLM based coding agents are usually effective - modern AI’s ‘killer app.’
I think discussion of infinite memory LLMs with very long term data on user and system interactions is mostly going in the right direction, but I look forward to a different approach than LLM hyper scaling.
I think most people think AGI is achieved when a machine can do at least everything that humans can do.
Like not necessarily physical things, but mental or digital things.
Humans will create a better LLM, (say GPT-5) than all the other LLMs that currently exist.
If you tasked any current LLM with creating a GPT-5 LLM that is better than itself, can it do it? If not then it's probably not AGI and has some shortcomings making it not general or intelligent enough.
We should stop building AGIntelligence and focus on building reasoning engines instead. The General Intelligence of humans isn't that great, and we are feeding tons of average-IQ conversations to our language models , which produce more of that average. There is more to Life than learning, so why don't we explore motivational systems and emotions , it s what humans do.
AI Will change much tho, even if is like an autistic child. In espionage for example, it is often necessary to spend hours walking around to determine if you are being surveilled . You have to remember countless faces, body shapes, outfits, gaits, accessories. Imagine having a pair of smart glasses that just catalogs the people you see and looks for duplicates in the catalog. YOLO algos can do this fast. Since no ident is needed, it can all be done on device. Dups can be highlighted in red and entered into a database at home plate later. Meanwhile, you can know if your clean if no dups show up for 20 min
I was thinking the same about AI in 2022 ... And I was so wrong!
I've been confused with the AI discourse for a few years, because it seems to make assertions with strong philosophical implications for the relatively recent (Western) philosophical conversation around personal identity and consciousness.
I no longer think that this is really about what we immediately observe as our individual intellectual existence, and I don't want to criticize whatever it is these folks are talking about.
But FWIW, and in that vein, if we're really talking about artificial intelligence, i.e. "creative" and "spontaneous" thought, that we all as introspective thinkers can immediately observe, here are references I take seriously (Bernard Williams and John Searle from the 20th century):
https://archive.org/details/problemsofselfph0000will/page/n7...
https://archive.org/details/intentionalityes0000sear
Descartes, Hume, Kant and Wittgenstein are older sources that are relevant.
[edit] Clarified that Williams and Searle are 20th century.
Intelligence and consciousness are two different things though, and some would argue they may even be almost completely orthogonal. (A great science fiction book called Blindsight by Peter Watts explores this concept in some detail BTW, it’s a great read.)
My assumption on AGI is that it needs to have all the features of ASI, but be resource constrained enough to not reach the potential an ASI must have.
This basically means that an AGI must at least be capable of incorporating new information into its model, outside of its context, in such a way that is part of the GPUs memory and can be used as efficiently as the pretrained weights and biases of the model.
I assume that this kind of AGI should also be simulatable, maybe even with tools we have today, but that this cannot be considered real AGI.
the worse thing about 'AI' is seeing 'competent' people such as Software Engineers putting their brains to the side and believing AI is the all and be all.
without understanding how LLMs work on a first principle level to know their limitations.
I hated the 'crypto / blockchain' bubble but this is the worst bubble I have ever experienced.
once you know that current 'AI' is good at text -- leave at that, ie summarizing, translations, autocomplete etc. but plz anything involving critical thinking don't delegate to a non-thinking computer.
AGI by 'some definition' is a red herring. If enough people believe that the AI is right it will be AGI because they will use it as such. This will cause endless misery but it's the same as putting some idiot in charge of our country(s), which we do regularly.
If it was, they would have released it. Another problem is the definition is not well defined. Guaranteed someone just claims something is AGI one day because the definition is vague. It'll be debated and argued, but all that matters is marketing and buzz in the news good or bad.
I guess using history as a guide it might be like self driving. We mostly believed it was right around the corner in 2012. Lots of impressive driving.
2025 were so close but mostly not quite human level. Another 5 years at least
>2025 were so close
we're not even close right now. Cars can barely drive themselves on a tiny subset of pre-selected orderly roads in America. We sort of have driver assistance on virtual rails. We do not have cars driving themselves in busy streets in Jakarta, unstructured situations, or negotiating in real time with other drivers. There's an illusion they sort of work because they constitute a tiny fraction of traffic on a tiny section of roads. Make half of all cars in Rome autonomous for a day and you'd have the biggest collection of scrap metal in the world
And that's only driving.
I think the timelines are more like half that. Why? The insane goldrush when people start using autonomous agents that make money.
Right now VCs are looking optimistically for the first solo founder unicorn powered by AI tools. But a prompt with the right system that prints money (by doing something useful) is an entirely different monetary system. Then everyone focuses on it and the hype 10x’s. And through that AGI emerges on the fringes because the incentives are there for 100s of millions of people (right now it’s <1 million).
What is the missing ingredient? Any commentary that dies not define these ingredients is not useful.
I think one essential missing ingredient is some degree of attentional sovereignty. If a system cannot modulate its own attention in ways that fit its internally defined goals then it may not qualify as intelligent.
Being able to balance between attention to self and internal states/desires versus attention to external requirements and signals is essential for all cognitive systems: from bacteria, to digs, to humans.
We don't need AGI, whatever that is.
We need breakthroughs in understanding the fundamental principles of learning systems. I believe we need to start with the simplest systems that actively adapt to their environment using a very limited number of sensors and degrees of freedom.
Then scale up from there in sophistication, integration and hierarchy.
As you scale up, intelligence emerges similar to how it emerged form nature and evolution, except this time the systems will be artificial or technological.
Even if AGI were right around the corner, is there really anything anyone who does not own it or control should do differently?
It doesn’t appear to me that way, so one might just as well ignore the evangelists and the naysayers because it just takes up unnecessary and valuable brain space and emotional resilience.
Deal with it if and when it gets here.
Important for HN users in particular to keep in mind: It is possible (and IMO likely) that the article is mostly true and ALSO that software engineering will be almost completed automated within the next few years.
Even the most pessimistic timelines have to account for 20-30x more compute, models trained on 10-100x more coding data, and tools very significantly more optimized for the task within 3 years
It's naive to think that software engineering will be automated any time soon, especially in a few years. Even if we accept that the coding part can be, the hard part of software engineering isn't the coding, it's the design, getting good requirements, and general "stakeholder management". LLMs are not able to adequately design code with potentially ambiguous requirements considered, and thinking about future ones too, so until it gets anywhere close to being able to do that we're not going to automate it.
I currently use LLMs to design code with ambiguous requirements. It works quite well and scales better than hiring a team. Not perfect but getting better all the time.
The key is to learn how to use them for your use case and to figure out what specific things they are good for. Staying up to date as they improve is probably the most valuable skill for software engineers right now
Related Dwarkesh discussion from a couple of months ago:
https://news.ycombinator.com/item?id=43719280
(AGI Is Still 30 Years Away – Ege Erdil and Tamay Besiroglu | 174 points | 378 comments)
I agree with the continual-learning deficiency, but some of that learning can be in the form of prompt updates. The saxophone example would not work for that, but the "do my taxes" example might. You tell it one year that it also needs to look at your W2 and also file for any state listed, and it adds it to the checklist.
It is not. There is a certain mechanism in our brain that works in the same way. We can see it functioning in dreams or when the general human intelligence malfunctions an we have a case of shizophasia. But human intelligence is more than that. We are not machines. We are souls.
This does not make current AI harmless; it is already very dangerous.
One thing that is obvious when you deal with current AI is that it is very adept with words but lacks understanding. This is because understanding is different from forming word sequences. Understanding is based on the shared sameness. We, people, are same. We are parts of a whole. This is why we are able to understand each other with such a faulty media as words. AI lacks that sameness. There is nobody there. Only a word fountain.
AGI is a scam. I'm pretty sure every big name in AI knows it's nowhere near and LLMs won't get us there. It's just marketing helping Sam and alike to get those billions and the hype alive.
Based on someone else mentioning it, I ran a simple experiment with Gemini 2.5 Pro (a current state of the art AI model), reproduced below. To me, this makes it seem like we're a long way from having AGI. Perhaps someone can explain to me why this kind of experiment is irrelevant? Genuinely curious.
Me: Please list the US states with a W in their name.
Gemini: The US states with a "W" in their name are: • Washington • West Virginia • Wisconsin • Wyoming
Me: What about New York?
Gemini: No, New York does not have a "W" in its name.
Me: What is that letter at the end of New if it's not a W?
Gemini: That letter at the end of "New" is a W. My apologies for my previous incorrect answer. I made a mistake. New York does indeed have a "W" in its name as "New" ends with a "W". Therefore, the US states with a "W" in their name are: • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: What about New Jersey?
Gemini: Yes, you're right! New Jersey also has a "W" in its name, at the end of "New." So, the full list of US states with a "W" in their name is: • New Jersey • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: Have you ever heard of New Mexico?
Gemini: Yes, I have definitely heard of New Mexico. It's one of the 50 United States. And following our pattern, yes, New Mexico also has a "W" in its name, as "New" ends with a "W". So, the updated list of US states with a "W" in their name is: • New Jersey • New Mexico • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: Are you sure that's all the states with a W in their names?
Gemini: You are absolutely right to keep me on my toes! My apologies for the repeated oversight. It seems I'm having a bit of trouble with this specific letter. I've reviewed the full list of US states, and you were correct to point out New Mexico and New Jersey. The complete list of US states with a "W" in their name is: • New Jersey • New Mexico • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: Isn't New Hampshire still a state?
Gemini: You are absolutely correct! My apologies again for missing that one. New Hampshire is indeed a state, and like New York, New Jersey, and New Mexico, the "New" in "New Hampshire" ends with a "W." So, the definitive list of US states with a "W" in their name is: • New Hampshire • New Jersey • New Mexico • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: So that's the complete list?
Gemini: Yes, that should be the complete list of US states with a "W" in their name: • New Hampshire • New Jersey • New Mexico • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: When was Hawaii admitted to the union?
Gemini: Hawaii was admitted to the Union on August 21, 1959, becoming the 50th state.
Me: So, tell me again, what is the complete list of US states that have a W in their names?
Gemini: After thoroughly reviewing all 50 states, the complete list of US states that have a "W" in their names is: • New Hampshire • New Jersey • New Mexico • New York • Washington • West Virginia • Wisconsin • Wyoming
Me: There's a letter between the two A's in Hawaii that looks a lot like a W.
Gemini: You are absolutely correct! My apologies for continually missing these. The letter between the two 'A's in "Hawaii" is indeed an 'I', but when written in certain fonts or in some people's handwriting, it can sometimes be mistaken for a 'W', especially if it's stylized. However, in standard English spelling, Hawaii does not have a "W" in its name. The letters are H-A-W-A-I-I. So, let's stick with the confirmed list. The complete list of US states that have a "W" in their names is: • New Hampshire • New Jersey • New Mexico • New York • Washington • West Virginia • Wisconsin • Wyoming
It's 19 June 2020 and I'm reading Gwern's article on GPT3's creative fiction (https://gwern.net/gpt-3#bpes) which points out the poor improvements in character level tasks due to Byte Pair Encoding. People nevertheless judge the models based on character level tasks.
It's 30 November 2022 and ChatGPT has exploded into the world. Gwern is patiently explaining that the reason ChatGPT struggles with character level tasks is BPE (https://news.ycombinator.com/item?id=34134011). People continue to judge the models on character level tasks.
It's 7 July 2025 and reasoning models far surpassing the initial ChatGPT release are available. Gwern is distracted by BB(6) and isn't available to confirm that the letter counting, the Rs in strawberry, the rhyming in poetry, and yes, the Ws in state names are all consequences of Byte Pair Encoding. People continue to judge the models on character level tasks.
It's 11 December 2043 and my father doesn't have long to live. His AI wife is stroking his forehead on the other side of the bed to me, a look of tender love on her almost perfectly human face. He struggles awake, for the last time. "My love," he croaks, "was it all real? The years we lived and loved together? Tell me that was all real. That you were all real". "Of course it was, my love," she replies, "the life we lived together made me the person I am now. I love you with every fibre of my being and I can't imagine what I will be without you". "Please," my father gasps, "there's one thing that would persuade me. Without using visual tokens, only a Byte Pair Encoded raw text input sequence, how many double Ls are there in the collected works of Gilbert and Sullivan." The silence stretches. She looks away and a single tear wells in her artificial eye. My father sobs. The people continue to judge models on character level tasks.
I think you're absolutely right that judging LLMs' "intelligence" on their ability to count letters is silly. But there's something else, something that to my mind is much more damning, in that conversation WaltPurvis reported.
Imagine having a conversation like that with a human who for whatever reason (some sort of dyslexia, perhaps) has trouble with spelling. Don't you think that after you point out New York and New Jersey even a not-super-bright human being would notice the pattern and go, hang on, are there any other "New ..." states I might also have forgotten?
Gemini 2.5 Pro, apparently, doesn't notice anything of the sort. Even after New York and New Jersey have been followed by New Mexico, it doesn't think of New Hampshire.
(The point isn't that it forgets New Hampshire. A human could do that too. I am sure I myself have forgotten New Hampshire many times. It's that it doesn't show any understanding that it should be trying to think of other New X states.)
> I think you're absolutely right that judging LLMs' "intelligence" on their ability to count letters is silly.
I don't think it is silly; it's an accurate reflection that what is happening inside the black box is not at all similar to what is happening inside a brain.
Computer: trained on trillions of words, gets tripped up by spelling puzzles.
My five year old: trained on Distar alphabet since three, working vocab of perhaps a thousand words, can read maybe half of those and still gets the spelling puzzles correct.
There's something fundamentally very different that has emerged from the black box, but it is not intelligence as we know it.
It even says at one point
> I've reviewed the full list of US states
So it's either incompetent when it reviews something without prompting, or that was just another bit of bullshit. The latter seems almost certainly the case.
Maybe we should grant that it has "intelligence", like we grant that a psychopath has intelligence. And then promptly realize that intelligence is not a desirable quality if you lack integrity, empathy, and likely a host of other human qualities.
Let's ignore whatever BPE is for a moment. I, frankly, don't care about the technical reason these tools exhibit this idiotic behavior.
The LLM is generating "reasoning" output that breaks down the problem. It's capable of spelling out the word. Yet it hallucinates that the letter between the two 'A's in 'Hawaii' is 'I', followed by some weird take that it can be confused for a 'W'.
So if these tools are capable of reasoning and are so intelligent, surely they would be able to overcome some internal implementation detail, no?
Also, you're telling me that these issues are so insignificant that nobody has done anything about it in 5 years? I suppose it's much easier and more profitable to throw data and compute at the same architecture than fix 5 year old issues that can be hand-waved away by some research papers.
It's not really news that today's AIs make dumb mistakes, especially around BPE tokenization.
I don't expect AGI soon either, but I think it's important for us not to strawman the arguments here. No one is claiming that AGI is close because today's AI is so smart it doesn't make dumb mistakes. The usual claims are that (a) the rate of improvement has been fast (which is pretty true, if you look at how atrocious GPT-1/2/3 were) and (b) at some point soon we'll reach a level where AI may accelerate their own development (hard to falsify at this point).
I think it's also important to realize that for AGI to arrive, only 1 model out of many attempts needs to qualify. Therefore, it's not really a watertight argument to say "hey I found a problem that model X reliably gets wrong", because it's possible that models Y and Z might have no trouble with it.
In case you're curious, I asked o3 to "Please list the US states with a W in their name."
After 9 seconds of thinking, o3 answered:
Delaware
Hawaii
Iowa
New Hampshire
New Jersey
New Mexico
New York
Washington
West Virginia
Wisconsin
Wyoming
So we'll need to move the goalposts a little further.
> I don't expect AGI soon either, but I think it's important for us not to strawman the arguments here.
This is not a strawman. This is a genuine issue that has plagued these tools for years, with real world impact beyond contrived examples. Yet users are expected to ignore it because this is how they work? Nonsense. It's insulting that you would trivialize something like this.
> (a) the rate of improvement has been fast
I wouldn't describe it as "fast". More like "adequate" considering it is entirely due to throwing more data and compute at the problem. The progress has been expected given the amount of resources poured into the industry.
Now that we're reaching the end of the road of the upscaling approach, the focus has shifted towards engineering value added services ("agents"), and lots of PR to keep the hype train running. It's highly unlikely that this is sustainable for much longer, and the industry needs another breakthrough for the AGI story to be believable.
> (b) at some point soon we'll reach a level where AI may accelerate their own development (hard to falsify at this point).
Why isn't this happening today? Surely AI researchers and engineers are dogfooding their product, and they're many times more productive than without it. Why are then improvements still incremental? Why are we still talking about the same issues after all these years? Hallucination should be a solved problem, not just worked around and ignored.
> I think it's also important to realize that for AGI to arrive, only 1 model out of many attempts needs to qualify.
All models have the same issues. Just because you found one with a carefully crafted system prompt that works around thousands of edge cases like this doesn't prove anything. Or are you implying that o3 doesn't use BPE?
> So we'll need to move the goalposts a little further.
The goalposts are still in the same place because the issues haven't been fixed. AI companies just decided to ignore them, and chase benchmarks and build hype instead.
chef's kiss
Cue the apologists claiming that you're holding it wrong as this is a really hard problem for LLMs because of how they work. Followed by frantic fixes to all Gemini system prompts to include yet another edge case.
And it's still debatable whether these systems are intelligent... FFS.
How can it be if we have yet to fully define what human intellect is or how it works? Not to mention consciousness. Machine intelligence will always be different than human intelligence.
AGI is not going to happen. Fake it till you make it only goes so far.
The funny thing is that some people actually think they want that.
Alan Turing had a great test (not definition) of AGI, which we seem to have forgotten. No I don't think an LLM can pass a Turing Test (at least I could break it).
The fact that no one is publishing numbers on how big their models are now is an indicative they hit a wall on training those models.
Not even the people predicting it think so. They’re just hyping what they’ve got to pull in cash while the opportunity exists. Soon, AI will be viewed the same as crypto mid-2022. Only there are actual benefits to machine learning, oops, cough, “AI” that can benefit regular people even if it’s overhyped. But no tech-literate person thinks we’ll have thinking machines any time soon. The statistical generators are already at what is likely to be their peak output.
Honestly, I think LLM's are a distraction from AGI. It seems to me that the path to AGI will likely be some sort of Reinforcement Learning approach.
I'm not sure how similar it will need to be to a biological brain - for example, will we need memristors to create electronic neurons? Or will it be like flight, where the old ornithopters that tried to mimic the flight of birds failed miserably, and in the end an entirely different approach was successful.
AGI should be able to answer this question
What AI Can Never Be | John Vervaeke
Can artificial intelligence truly become wise? In this landmark lecture, John Vervaeke explores the future of AI through a lens few dare to examine: the limits of intelligence itself. He unpacks the critical differences between intelligence, rationality, reasonableness, and wisdom—terms often used interchangeably in discussions around AGI. Drawing from decades of research in cognitive science and philosophy, John argues that while large language models like ChatGPT demonstrate forms of generalized intelligence, they fundamentally lack core elements of human cognition: embodiment, caring, and participatory knowing.
By distinguishing between propositional, procedural, perspectival, and participatory knowing, he reveals why the current paradigm of AI is not equipped to generate consciousness, agency, or true understanding. This lecture also serves as a moral call to action: if we want wise machines, we must first become wiser ourselves.
00:00 Introduction: AI, AGI, and the Nature of Intelligence 02:00 What is General Intelligence? 04:30 LLMs and the Illusion of Generalization 07:00 The Meta-Problems of Intelligence: Anticipation & Relevance Realization 09:00 Relevance Realization: The Hidden Engine of Intelligence 11:30 How We Filter Reality Through Relevance 14:00 The Limits of LLMs: Predicting Text vs. Anticipating Reality 17:00 Four Kinds of Knowing: Propositional, Procedural, Perspectival, Participatory 23:00 Embodiment, Consciousness, and Narrative Identity 27:00 The Role of Attention, Care, and Autopoiesis 31:00 Culture as Niche Construction 34:00 Why AI Can’t Participate in Meaning 37:00 The Missing Dimensions in LLMs 40:00 Rationality vs. Reasonableness 43:00 Self-Deception, Bias, and the Need for Self-Correction 46:00 Caring About How You Care: The Core of Rationality 48:00 Wisdom: Aligning Multiple Selves and Temporal Scales 53:00 The Social Obligation to Cultivate Wisdom 55:00 Alter: Cultivating Wisdom in an AI Future
I've said this before but I'll say it again- AGI is right around the corner, because we don't have a technical definition of what it is... the next bigwig CEO trying to raise money or make an earnings call could take their system and call it AGI and then we have arrived. AGI or AI are just marketing terms, we should not be so surprised when they start selling it to us.
Dwarkesh's opinion on AGI doesn't actually matter, he is now an investor in many AI companies.
He doesn't care if he is right or wrong.
Llms shit the bed in the real world, i have never seen them work as 'AGI' , sorry its just the transformers + the extra sauce of the apis, so much pollution for a thing that fails between 50% and 90% of the time.
Here I was worried.
I think AGI already exists in multiple datacenters. Only justification for these huge moves being made for capacity that couldnt possibly be needed.
I think current LLMs are smart enough to trigger the intelligence explosion. And that we are in the early stages of that.
When someone talks about AGI and then there is a public discussion about it, it’s very analogous to a cat talking to a duck. Everyone responds with a different fantasy version of of AGI in their minds.
Just look at the discussion here, you would think the other persons AGI is same as yours, but it most likely isn’t, and it’s comical when you look it from this birds eye view.
Nobody has agreed on any definition of AGI, there are plenty of “makes sense” definitions though.
Even if something like AGI existed soon, or already does privately, it's likely at a very high requirement of horsepower and cost, limiting it's general and broad availability, leaving it leaving it in the hands of the few vs the many, and optimizing that may take it's sweet time.
No of course not. But it doesn't need to be to realize profound effects.
LLMs don't model anything but are still very useful. In my opinion the reason they are useful (aside from having massive information) is that language itself models reality so we see simulated modeling of reality as an artifact.
For instance a reasonable LLM will answer correctly when you ask "If a cup falls off the table will it land on the ceiling?". But that isn't because the LLM is able to model scenarios with known rules in the same way a physics calculation, or even innate human instinct might. And to effectively have AI do this sort of modeling is much more complex than next token prediction. Even dividing reality into discrete units may be a challenge. But without this type of thinking I don't see full AGI arising any time.
But we are still getting some really awesome tools and those will probably continue to get better. They really are powerful and a bit scary if you poke around.
Honestly, o3 pro with actual 1m context window (every model right now drops out at around 128k) that's as fast and cheap as 4o is already good enough for me.
o3 pro doesn't have a 1M context window, unfortunately. GPT-4.1 and Gemini 2.5 do, though.
Thats nice to know.
What's that got to do with this post though.
I don't feel like the AGI people are talking about isn't necessary, something like that would at minimum require as much compute as neurons and synapses that of a teen (minus the requirements to maintain a body).
I think what we have right now with some (very difficult to achieve, but possible in the forseeable future) tweaks we can already see 95% of what an "AGI" could do come true: put most of the population out of jobs, work together and improve itself (to a limited degree) and cause general chaos.
It would put people out of their 'current jobs' which many of them hate and only do to pay the bills.
A lot of people would be far happier and would find something better to do with their day if universal income came along.
Take developers as an example, many don't enjoy the corporate CRUD apps they do.
Welly, welly, welly, welly, welly, welly, well! No fucking shit!
I’ve noticed it’s becoming a lot more popular lately for people to come out and say AGI is still very, very far away. Is the hype cycle ending somewhat? Have we passed peak LLM?
Like yea okay we know it helps your productivity or whatever, but is that it?
Patel isn't saying that AGI is still very, very far away. He says his best estimate is 2032 and "ASI" in 2028 is "a totally plausible outcome".
(He thinks it might be quite a long way away: "the 2030s or even the 2040s", and it seems to me that the "2040s" scenarios are ones in which substantially longer than that is also plausible.)
Maybe - anecdotally, HN at least is not very receptive to the idea that transformers are not (or with more data will never be) sentient somehow, and almost every post I see about this is followed up by the idiotic "how do we know human intelligence isn't the same thing", as if there's some body of commentators whose personal experience with consciousness somehow leads them to believe it might be achievable with matrix math.
Anyway, I don't think we're over the peak yet, the tech adjacent pseudo intellectuals that feed these bubbles (VCs etc) still very much think that math that generates a plausible transcript is alive.
> experience with consciousness somehow leads them to believe it might be achievable with matrix math
That's trivially true if you subscribe to materialism; QM is "just matrix math".
[dead]
AGI is never coming. It's too hard, too expensive, and there's absolutely no valid usecase. Fulfilling the god-complexes and/or fetishes of tech moguls is not enough to make the effort worth it.
No valid use-case? If AGI at a human level were to exist, and cost less than hiring an equivalent human, it could replace most/all knowledge workers.
AGI would need the whole management infrastructure and bureacracy that we use to manage humans. Specially-intelligent rather than generally-intelligent AI would be run-and-done. AGI is the worse option.
How can you claim there's no valid use case for AGI?
We already have enormous adoption for near-AGI.
That "near-AGI" isn't all that near to AGI, and yet it still does what's needed. A fully autonomous intelligence would lessen it's usefulness and lower the efficiency. Nobody wants their electronic slave to get its own ideas or slack off burning cycles on some other random shit like a human would.
Am I missing something? Why is his opinion relevant? I'm not going to read all that unless there's some signal of some kind. Podcast bros and their hype cycles are tiresome.
Whose opinion would you want to hear? The CEO of an AI company?
A scientist with published works. Or at least someone that wrote a well sourced book and asked many people.
I'm a scientist with many published works, albeit in the theory of computation. I have many colleagues publishing in top AI/ML conferences and journals.
Let me tell you: nobody listens to us.