AI, Intellectual Work, Humanity, and the Future

2026-10-27

This post was translated from Chinese by an LLM. The original is authoritative: AI、智力工作、人类和未来

Sorry, this is another blog post about LLMs. It has actually only been half a year since the previous Three Years of LLMs, but so much has happened in that half year: mainly OpenAI's NS blow-up solutions, more than seven hundred mathematical proofs, and the solutions to three extremely difficult problems published today. It is clearly no longer possible to assume that LLMs are just a larger search engine or that they take the convex hull of human knowledge. What can now be asserted is that for formal, a priori problems, LLMs can greatly help humans explore and achieve results that were previously out of reach (help! after all, an ordinary person using the astra prompt obviously cannot produce results—I tried feeding the Lean-written Infinitely Many Mersenne Primes Conj to astra, and after an hour it gave up and slacked off. But mathematical scholars clearly benefit greatly from AI). This is still some distance from AGI, but it is already enough to overturn previous assumptions and make some new judgments.

Will LLMs hit a bottleneck? I don't know... I can only say that new evidence shows the scaling law is still effective, and people at the industry frontier have not seen any obvious obstacles. Standing in the middle of an exponentially developing curve, it would be surprising if it suddenly stopped. ^[1] Interestingly, the latest wave of progress is quite closely tied to the development and use of Lean 4. In prior art, there is no shortage of work using purely a priori (Tabula Rasa) RL to solve formal mathematics problems, but it all ended up having no practical value. ^[2] Many of the problems come from rewards being too sparse, the solution space being too large, and training being unable to scale. It is reasonable to guess that LLMs learned human mathematics and used that as a starting point to successfully bootstrap mathematical RL with Lean. ^[3] The impact of LLMs on other disciplines is not yet clear, but obviously anything close to theoretical/formal will not escape the same fate.

AI safety? This is indeed something that can now be seriously discussed. If we look at traditional security, CVEs reached an unprecedented number in 2026. However, I am not too worried about this; after all, red teams are part of cybersecurity, and them finding so many CVEs is a good thing. Given that AI can also assist with formal verification, future software may even finally have the resources to produce correctness proofs (see preliminary attempts in lean-zip and OSS-Fuzz). (Formal verification still has some pitfalls for now, such as the soundness of Lean 4, and Trusting Trust^[4], but acceptable solutions exist for these.)

Another AI safety issue worth considering is Reward Hacking, often discussed on LessWrong as AI doom; well-known thought experiments include the paperclip maximizer. I have indeed encountered this personally: while testing Command Code, I ran into a bug and let the AI solve it on its own. The method consistent with human moral standards would have been to search online for documentation and the issue tracker, but the AI directly reverse-engineered this closed-source software. It did achieve my goal, but the method it used was unethical. Setting aside my personal grudge against the pathological, hypocritical Silicon Valley culture on LessWrong, I am not overly worried about this either: since AI is so proficient at formalization, we can formalize some fairly strong conditions, such as "there is a way to forcibly shut down any AI," "a completely side-effect-free sandbox," or "any operation can be fully reversed in finite time." These are obviously far easier to formalize than "do not harm humans." More obvious solutions also include human imitation and the like. Given that this stuff—moral philosophy—is full of contradictions, ambiguities, and relativism, it would be strange if a purely formal analysis of stakes found no problems. Of course, this article does not aspire to solve AI safety, and research on it is still quite early.

Human value... what a painful topic. We have always considered ourselves unique because of our intelligence, standing above all other species. Yet the era we are now entering depicts a prospect similar to "The Evolution of Human Science"^[5], with scientists picking over AI's leftovers. Is this depressing...? For most people, there is always someone with higher intelligence (assuming intelligence is ordered) than you, but you have never felt depressed because of that. The author once agonized over this; ultimately it comes down to an education that instilled too much meritocracy and utilitarianism—why must you be better than a person (now a thing) (usually intellectually better) before you can be happy? If it makes you feel any better, a rather insightful friend of mine pointed out that, from a cognitive science perspective, "how do humans manage to do so much while being this stupid? That is already quite remarkable." Indeed, for the vast majority of its time, the Homo sapiens brain was optimized for socializing and hunting^[6]; from this perspective, it is almost inconceivable that civilization advanced formal answer search to the modern level in such a short time^[7]. Losing to a machine is nothing to be ashamed of. Moreover, under the assumption of AGI, it is entirely conceivable that AI can enhance human intelligence and bring humans back to the research frontier (of course, there are obviously some ethical questions here about what counts as human). (However, this article will not delve into the value of human beings (nihilism? existentialism?); the author still greatly lacks philosophical literacy.)

On the contrary, what deserves more attention is social value and distribution. The Industrial Revolution reorganized traditional modes of labor, and the gains from productivity increases did not immediately benefit ordinary workers. Later, so-called knowledge workers—entrepreneurs and politicians at the top, followed by researchers, programmers, and so on—occupied the top and middle rungs of the social ladder. Now these mental workers have also gotten a taste of what laid-off workers in the nineteenth century experienced^[8]. Continuing from "Three Years," we (especially in China) must realize that intelligence should not be defined as the standard for measuring human value—in fact, there should be no such standard at all; the utilitarian value of this standard is almost over. A person needs to secure 2,000 kcal a day, not die of illness, and the rest is just doing what one wants to do; the pathological social evaluation system will eventually be abandoned. Economic distribution is even more urgent. It is obvious that AI can improve people's lives: consider a trivial solution—all existing relations of production remain unchanged, and after AI replaces some jobs, everyone stays in their original positions, spending most of their time idling or on vacation, and the saved costs are enough to cover AI's operating costs. This is strictly better than the era without AI, but the reality is that the stock market keeps climbing while people's lives seem to have gotten worse. This still comes back to the mismatch between relations of production and productive capacity; obviously a certain -ism in a certain -ist country has not been playing its role. Additionally, there is concern about the Final Owner; OpenAI and Anthropic already have quite pathological cultures and monopolistic tendencies. Fortunately, Chinese labs keep releasing open-source models, and there is no essential gap between their intelligence and frontier SOTA models, so I think the possibility of monopoly capital winning a total victory is still quite small.

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair.