Technological Speed Limit

What happens when you saturate the learning curve?

Moore’s Law held from when he identified the trend in 1965 until around 2016 when our ability to make ever-smaller tranisters final abated. While hardware continues to get faster

My first computer was an Atari 800 but my family’s first PC was a clone by AST Research with a 10MHz 80286 processor. In those days every time the transistors got smaller, thanks to Moore’s Law, the clockspeed would be faster, because smaller transitors could be driven faster, until that ended around 2005.

After 2005 Moore’s Law continued to bring us smaller transisters until around 2012, when the improvements gently slowed. Today they are still slowly making transistors smaller, but today each wire holds

This came to an end around 2005 with clock speeds hovering around 3GHz. Moore’s Law continued, but now the “transistor budget” went into parallism at every level of the chip, where multiple cores were just one of many ways to spend transitors to make faster processors.

I pondered Moore’s Law for decades, as clockspeeds rose from 10MHz, my family’s first PC, all the way up to 3GHz. I read hundreds of articles about how while clock speeds would stop rising, due to unfixable thermal issues, Moore’s Law itself would continue. And and

I’d read a thousand articles on how amazing the exponential progress was while speculating on when it would top out. While clock speeds stopped rising around 2005, Moore’s Law itself continued, with increasing transistor density going into parallelism and other features. Eventually I found myself asking two different questions: why does the technology improve so steadily, and why doesn’t it improve faster?

One day walking our dog it hit me. Steady improvement over sixty years meant that there was a fundamental speed limit in how quickly we could improve the technology. There’s no other way to explain the steadiness except that we are bumping up against a limit. There’s no way we’d be balancing on a saddle point, just happening to hit the same rate of improvement year after year. If we were not at a limit, you’d expect to see periods of stagnation and acceleration at every scale, slow years, slow decades, etc. Instead, despite a never-ending parade of companies, individuals, approaches and funding cycles, improvements come like clockwork because we are riding a limit.

I realized our ability to improve the technology has a maximum theoretical “learning rate” and we cannot improve faster than that rate. The reason for this, I surmised, is the entire industry is a complex physical system, and I’m including humans and their human brains in the system. No matter how hard you push on the system, it will only unspool new technologies at a fixed maximum rate. The reason is that the system as a whole cannot “skip steps”. We invented GPUs to play video games, and we wrote video games to take advantage of new GPUs; this symbiosis has to “spin up” over time. If you went back to 1970 and said “create a GPU to play 3D games” it wouldn’t land, it was too early.

Where does this “technological speed limit” appear in today’s AI era? For starters I think it fully explains why the legacy tech giants, with seemingly unlimited resources, cannot catch Anthropic and OpenAI. Those two startups are well funded and well staffed enough to be operating at full speed, bumping up against the speed limit for improving AI. While companies like Google and even open source models can rival and at times equal them, no one can pull far ahead, because no one can consistently exceed the limit.

This isn’t the same speed limit that semiconductors hit, but there is a speed limit for the same reason: the physical system of people, brains, and markets can only progress so fast. The steadiness of the improvements in both chips and AI isn’t obvious because up close the industries are chaotic and varied, but zoom out enough and you see a smooth steady progression.

What about when AI starts doing AI research, which Andrej Karpathy is exploring with his AutoResearch and which every large AI company is surely doing as well. This is the big transition that’s meant to trigger an “intelligence explosion”.

One possibility is we are already a century into an intelligence explosion, and there’s no “other gear” that we’ll launch into even when AI is developing AI. The point of view here is that “just” keeping the same exponential progress going will require recursively using AI to keep things going. The other possibility is “this time it’s different” and things become dramatically super-exponential.

Here I look back over the last 60 years of economic activity where none of the miraculous breakthroughs we saw bust us out of our baseline 2-3% GDP growth. Robert Solow said, “You can see the computer age everywhere but in the productivity statistics”. Computers have gotten trillions of times faster, but there’s no commensurate increase in productivity. From this perspective while AI feels exotic and new, it might only keep the same trends going.

Perhaps “AI developing AI” is so radically different from everything that’s come before, that we’ll slide into a literal explosion of nearly instantaneous progress. But my strong feeling is no, because the small bits of extreme progress will be embedded in the much larger system of the full economy. Amdahl’s Law tells us if you make 50% of a system 1,000,000 times faster, the overall system only speeds up by 2x.

I believe with AI improving AI we will see these insane pockets of technological progress, but that will be just one change in a much larger system, the whole economy, such that it will “just” keep the current exponential improvements going, and not launch us into an era of hypergrowth.