A Workaphile’s Apology

Disclaimer: No LLMs were used in the writing of this essay—all em dashes are mine¹.

Soon after the AlexNet breakthrough burst onto the scene over a decade ago, I became preoccupied with the following question:

What will a meaningful human life look like when thinking is best done by machines?

Versions of this question are now frequently discussed in the mainstream². I have yet to, however, come across a satisfactory answer that is applicable to a particular kind of person (more on this shortly). My own wranglings with the question were fruitless for the longest time but during the last year I have begun to glimpse outlines of an answer. These outlines are blurry and will likely remain so for the immediate future, but I believe they are interesting enough to share.

The question as I posed it contains multiple ambiguous words and resolving these ambiguities is critical to answering it. One common thread in the public sphere has centered around what is meant by “thinking”, whether AGI can surpass human intelligence in its full generality, and whether such generalness even exists, questions that I find distressingly dull. While it is indeed very interesting that the main form of non-human intelligence (large language models or LLMs) we have so far built is seemingly radically different from our own, at least developmentally³, it is also clear that this intelligence is general and will soon match and exceed human intelligence across all dimensions, if it has not already⁴.

The ambiguous words in the question that I do find interesting are “meaningful”, “human”, and “machine”. We take for granted what “human” means, but I do not think we should. To start though, let us begin with “meaningful”, as meaning lies at the core of what I am trying to answer and is the aspect of human life that I believe will be most impacted by AGI.

Whither meaning?

There is, obviously, no single way to live a meaningful human life. Some ways may be more rewarding to the individual; some more beneficial to society; some lead to great material wealth; some to inner growth; all can bestow meaning on an individual when suited to their personality and preferences. It is safe to say that our ways of living have multiplied as the technological capabilities of our civilization have increased. I expect this to continue to be true after AGI. But a particular kind of way of life, and a particular kind of person, may not. This is the person whose primary source of meaning comes from their work. An excellent blog post by Harvey Lederman, a philosopher at NYU, nicely articulates the divide between those who derive meaning from their work and those who do not, and the way in which AGI will differentially impact people who do. While Lederman’s blog post is not required reading for this one, it does set the stage nicely.

Let’s call the people in question “workaphiles”.⁵ They enjoy their work to the degree that they have trouble defining boundaries with other aspects of their lives, and they view their life’s primary purpose as that of doing useful work. Some may be maladjusted, even obsessive about their work, but that is not the defining trait in question.

Most people do not fit these criteria; they instead find meaning in raising a family, in adventuring, in the company of friends, in travel, in altered states of mind, in competitive sports, and so on. Those are perfectly valid ways to live a meaningful life and will likely become even more so in a post-AGI world. I would guess that most people (80-95%?) do not find meaning in work yet currently have to work to survive. If we safely and successfully transition to a post-required-work, post-AGI society, this large majority will be better off, living meaningful lives without the drudgery of non-purposeful work.

If you are not a workaphile, you may essentially ignore the rest of this blog post and await the upcoming utopia; you only have the questions of alignment—which I will not touch upon—and civilizational course—which I will—to worry about.

My concern is for workaphiles, as they are my tribe, and I do not want to see my people die out. We are not better people; many would say we are worse, and that is fine, but not so much so that we deserve to go extinct. I also think that workaphiles have punched above their weight in their contributions to human society in the past and they may yet have a role to play in its future. Lederman wistfully acknowledges our likely extinction and leaves it at that. I would like to see if there is something we can do about it.

A thought experiment to determine if you are a workaphile: imagine that the technology exists to make a perfect clone of yourself, including your present memory, mental state, and aptitudes. This clone can therefore do everything you can do. One of you will live the life you originally intended, carrying out useful work, ensuring that your net impact on society remains unchanged. The other gets to spend life in leisure and play. Without incurring any moral cost, you get to pick which of you lives the play-life and which lives the work-life. Which life would you pick?

Useful work

I just introduced another ambiguous concept: “useful work”. Being precise here is important as usefulness is central to the post-AGI conundrum. I take useful work to mean any activity that improves the happiness and well-being of others, present and yet-to-be. If what you do only makes you happier, then it is not useful work as I am defining it. Informally, it is certainly useful to make oneself happier—you too count—but my intent is to draw a distinction between activities that only empower individuals versus those that broadly lift society. One can also quibble over whether kin (and friends) count, and thus whether raising a family is useful work. I certainly think it is, but for our purposes here, it is simpler to focus only on work that helps non-kin. On the other hand, I am expansive in my definition of “others”, and include humans, animals, and sentient machines and extraterrestrial life.

At the extremes, it is easy to differentiate between useful and non-useful work. Erecting an apartment building is useful to all the people who get to live in it. Playing a videogame is perhaps less so⁶. The middle ground is where it gets murky. What about math, for instance? The answer might seem obvious (useful), but G. H. Hardy, a highly esteemed early 20^th century mathematician, wrote a whole book arguing for the uselessness of mathematics. In A Mathematician’s Apology (“apology” in the sense of apologia, i.e., in defense of something), Hardy asserted that mathematicians ought to do (pure) mathematics for its own sake, and not for some (un)anticipated future application⁷. A similar type of argument can be made for art; there is “useful” art that entertains crowds, communicates a political message, or lifts the mood of a nation, and there is “useless” art that embodies and advances its creator’s creative expression.

I describe these examples not to say that mathematics or art are not important—they very much are—but to emphasize that my definition of useful work is narrow, and it is this narrow version of work that I want to use because it is the one that will be most impacted by AGI. The practice of mathematics for self-fulfillment, as Hardy argued for, will not be affected by AGI in the way that “useful” mathematics, one that has eventual practical applications, will be (in fact, already has been). This splits workaphiles into two tribes: people who love their work because of the enjoyment they drive from it, irrespective of practical utility, versus people who also care about the broader impact. My concern is exclusively with the latter group. In this sense, my workaphile’s apology is the exact opposite of Hardy’s: he defended “useless” work; I’m defending “useful” work. A century can flip the script.

AGI will make useful people useless

We have people who drive meaning from useful work. I posit that AGI will make the work of these people useless, depriving them of their life’s purpose. By “people” I mean humans with a neurophysiology reflective of the species’ current genetic pool. This is subject to change but let us set that aside for a moment. Why do I think that useful work will no longer be usefully done by people?

The argument is simple (and frankly obvious) and has been cogently stated elsewhere so I won’t belabor it. Once machine intelligence exceeds human intelligence on every cognitive task, having a human in the loop will not only become unnecessary but detrimental, as humans’ meat brains will slow things down and introduce more errors. If the work is useful, its purpose is to elevate others rather than help the doer self-actualize, in which case it is best done most efficiently, i.e., by machines with silicon circuits or faster, future substrates.

There are important caveats worth noting. First, there is inertia against any transition. Humans and their institutions are slow to change, and so the role of the human orchestrator will likely persist for some time, for reasons of tradition, control, safety, legal liability, and uncertainty about AI performance. Second, some tasks are inherently about defining human values and needs. Individuals and societies will, hopefully, continue to decide what their priorities and objectives are at any given point in time. While some of this will be delegated to machines, at the most fundamental level humans will presumably still get to say I want to be happy or I want to know the truth. Third, some work, present and future, involves understanding the (human) mind’s inner states, as has been done for millennia by meditators within Eastern philosophical traditions. Almost by definition, this requires humans in the loop, at least until faithful simulacra are produced. Fourth, artistic and athletic performances and artifacts performed and made by humans may continue to hold sway, by virtue of their relatability, and would count as useful work if they lift people’s spirits. Fifth, certain kinds of work benefit from human contact, care, and emotional support, such as therapy and nursing, but I am less certain about this one because the human element can also introduce awkwardness and indignity. Sixth, a huge category that I defined out is the “work” of supporting and being in the company of kin and friends, which I suspect (and hope) will become one of our greatest preoccupations. That would likely lead to a better and more interconnected world. I failed it on a technicality because I know it is not the kind of work that I and other workaphiles care about, despite it being central to the human condition.

Beyond the above categories, and some others that I likely missed, the range of useful human work will narrow dramatically. The work will first change in nature, as has already happened to programming and is presently happening to mathematics, by moving up the abstraction hierarchy to the point of being unrecognizable to its prior practitioners, and becoming, to some⁸, no longer fulfilling. Then it will be extinguished entirely.

AGI will not make useful work unnecessary

At this point it may be tempting to imagine that soon after AGI becomes widely accessible, all useful work, not just that done by humans, will cease to be necessary, as AGI rapidly chews through our backlog of problems to render life as we know it solved and allow every person to kick back and enjoy the fruits of the species’ collective labor. While “soon” in the preceding sentence is doing a lot of heavy lifting, I suspect that whether it is for months, years, or decades after AGI, useful work will persist.

The reasons are multiple. To start with, we do indeed have a backlog of problems, relating to poverty, climate, energy, and disease. While AGI will invariably speed up progress, these problems are primarily ones of atoms, not bits, with their concomitant physical, chemical, and biological constraints. Animal models like mice are useful for testing and developing drugs but take months to grow. More faithful models like primates take even longer and raise serious ethical concerns. Unless we speed up the biological rate of animal development, or develop accurate simulations of human physiology, we will be bottlenecked by biology rather than intelligence. In due time, we will solve these problems, but there will be non-intelligence-related bottlenecks along the way.

Some problems also comprise complex systems that are inherently irreducible to simplifications and are thus not guaranteed to yield increased controllability given increased intelligence. Climate may be one such system. The human body another. Even if we were to develop methods for highly faithful simulations, running them at sufficient fidelity to model the effects of e.g., disease may be energetically expensive enough that it would present serious societal burdens. In due time, again likely solvable, but not overnight.

Here I find the concept of marginal returns to intelligence that Dario Amodei articulated in Machines of Loving Grace to be helpful. Namely, not all problems will benefit equally from increased intelligence (at least at first), and we should be looking for problems that maximize this property and prioritize them (more on this later).

Beyond the backlog, lies exploration, of the inner and outer kinds. Throughout our history we have not exclusively focused on the here and now, on our immediate problems. We have gone outwards, to new continents, moons, and planets, and inwards, to the inner worlds of our consciousness. There is no reason to believe we will want to stop after AGI. Exploring and colonizing other planets, searching for alien life, expanding the perceptual range of our sensory apparatus, inducing new mental states, and ultimately perhaps even facing the heat death of the universe—these all seem, for now, to be nearly inexhaustible reservoirs of useful future work.

So there will be work to be done; the question is: do we get to do it?

It is appropriate that I, before getting into the meat of the solution (no pun intended), offer my own apologia for why, in a world of abundance, a human would want to work. The crux of the matter for me personally is that I want to be where the action is, where the knowledge frontier is advanced. I cannot imagine a satisfying life where all scientific discoveries are made by machines, and we’re only told about them in ELIH⁹ fashion. I cannot imagine a satisfying life where new regions of space are explored, and we only get holographic feeds. I cannot imagine a satisfying life where an intelligent species is first contacted, but we are not the ones shaking their metaphorical hands. I cannot imagine a satisfying life that is lived vicariously, where every true achievement is done by a class of beings far beyond my own. I know I could play at science, play at discovery, play at exploration. But play is not enough. I know, in my deepest bones, that if I were to not do the thing, but have it be done for me, then my life would be devoid of true contentment.

Desiderata

Disclaimer: I am a molecular and systems biologist, not a neuroscientist. Take everything I say next with the requisite amount of salt that accompanies a lay person’s proposal. What I am proposing remains firmly in the realm of science fiction, but the world we are envisioning has abundant superintelligence, and so some blending of fiction into science is warranted.

To engage in useful work in the post-AGI era, humans will need to upgrade their brains so that they remain at, or can at least meaningfully engage with, the intelligence frontier. Before I get into the how, which will be vague because I don’t really know how, let me first describe what I see as the desiderata for post-AGI human brains, both in terms of end points and the process that gets us there.

Human brains are limited in speed, capacity, and upgradability, and these limitations affect both cognition (inference) and learning (training). These are the limitations we want to lift. We will want to increase in situ capacity, i.e., intelligence that is physically part of a person’s brain, either by increasing the brain’s volume or its computational density. We will also want to increase ex vivo capacity by enabling brains to communicate with external computational substrates, to transfer information (learning) and to integrate our cognition with larger computations (inference). Brains additionally encode identity and memory, and most importantly, are the seats of consciousness. Retaining these, at least as starting points, is a necessary feature.

Process-wise, we want to maximize gains while minimizing the risk of potential losses, for example loss of memory, or even worse, conscious experience, by ending up as philosophical zombies. Achieving this safely, I believe, will require a gradual improvement path so that brain transformations retain—or at least surface the degree of loss of—memory, identity, and conscious experience at every indivisible step of the process, permitting individuals to confirm the fidelity of their mental experience before and after every step.

Proposal

My core proposal is to replace biological neurons with synthetic ones, not at all once, but gradually, using engineered cells that transform natural neurons into more capable substitutes. A neuron’s input/output relationships, relative to all its incoming and outgoing synapses, would get mapped and replicated as faithfully as possible by the synthetic substitute. The degree of fidelity required will be a matter of research, but, given the centrality of the subjective experience in this process, development of the procedure would require objective and subjective tests that verify functional parity at the molecular and cellular levels as well as higher levels of learning and cognition, i.e., systems-level brain function. During early stages of this technology, replacements may occur one neuron at a time and aim for functional parity but with latent potential for greater capabilities, such as being faster, more plastic, and externally modulatable, and easier upgradability, providing future neuron replacement procedures with a foothold. As the technology matures, larger substitutions of bundles of neurons with defined synaptic perimeters can start getting made, either with one-to-one replacements of synthetic neurons, i.e., scaling the single neuron replacement technology, or with functional substitutes that have different internal computational structures but that retain the bundle’s original input/output behavior. Such larger deltas would permit greater flexibility in the replacements, and thus greater capacity for improvement, but require more comprehensive objective and subjective tests of cognitive and perceptual parity, respectively. Eventually, as ever larger replacements are tested, wholesale brain replacements could be completed.

End-to-end, the process would start with a biological human brain as we know it in 2026 and end with a conscious machine. The result would be a person that retains salient aspects of the purely biological human individual they once were but that can leverage the continued recursive advancements of machine intelligence. In the limit, such persons may become indistinguishable from their pure machine counterparts—ones without a human origin—but that would only occur through a willful process of transformation.

The technology required to achieve this proposal is so far from current capabilities that it’s hopeless to speculate on what would be required to achieve it, other than it would likely be an exercise in cellular engineering, synthetic biology, materials science, and, obviously, neuroscience. One wildcard here is AGI itself, and how much it can accelerate progress in those fields. I expect the near-term future of all science to shift from humans solving problems directly to setting up the harnesses required for machines to solve them, just as AlphaFold shifted the abstraction from modeling the phenomenon to building an algorithm that learns to do so. At least, until human-origin brains reclaim the intelligence frontier that we are now ceding.

Some alternatives

It is helpful to consider for a moment two competing alternatives that have been proposed and some of their comparative (dis)advantages.

Neuralink-style human augmentation: one common theme is augmentation of otherwise normal human brains through human-machine interfaces. I’m ascribing this to Neuralink only because it sounds similar to what they have publicly stated, although I have no idea what they are working on internally and it may well be much closer to my proposal. In this augmentation scheme, brains remain largely unaltered, including their internal processes, but get exposed through high-speed interconnects to digital data. This has the benefit of being comparatively less invasive, but the major drawback of ultimately relying on our meat brains to do all the internal processing. While we may be able to perceive and sense information far more quickly than we currently can, we would remain meat-in-the-loop, and that seems unlikely to be competitive for long. Sooner or later, this solution will run its course, quite possibly before it even gets going.

Kurzweil-style brain uploads: in The Age of Spiritual Machines, Ray Kurzweil popularized the idea of uploading human brains to digital clouds, where we can exist entirely free of biological limitations, essentially as pure information. This may ultimately be the terminal endpoint of what I am proposing, but as a starting step it has at least one serious problem. Absent a much better understanding of the physical nature of conscious experience, it will be impossible to know whether a transformation of the sort that Kurzweil envisages, assuming it is possible, would result in a conscious entity. We may face the same problem we face today with LLMs, where they appear rather conscious, but permit no mechanism of testing that appearance.

What I like about gradual cell-by-cell transformation is that it permits a continuous test of the conscious experience: we’ve replaced one of your neurons, do you feel any different? It tests the principle that consciousness is emergent out of the collective behavior of neurons and not any individual neuron. The process can go on and on, until all neurons are replaced, or nearly all, if the neural correlates of consciousness prove difficult to replicate using non-biological substrates. The size of the replacement deltas may obviously change, and perhaps one day we get good enough at this, and confident enough, that replacements are nearly instant and wholesale. But a gradual approach would allow us to test the science of consciousness with manageable risk.

One may object that a gradual process can fool the person into thinking they remain conscious while their conscious experience slowly slips away. This would argue for larger deltas, at least for early tests, to ensure that a sufficiently large perceptual change is encountered to trigger the person’s notice. Regardless of the precise size of change required, there is no reason to expect that one’s memory of the conscious experience is tied to one’s actual conscious experience. I.e., if a change were to occur in one’s consciousness, we would expect the person to notice. At least, this is the case for people who experience brain trauma, although not universally so.

Another advantage of such an approach is that it allows us, initially, to retain our everyday perceptual range, so that the change is never too jarring, while providing a pathway to developing brains that have, at their base level, the human experience, but that continue to advance at the pace of technological progress.

Form factors for brains

Even if we were to successfully upgrade our brains beyond their current limitations, and even if these upgrades streamline future upgrade cycles, there is no guarantee that human-origin brains will be able to reach the intelligence frontier. I see at least two major challenges, although it is likely that there will be more. The first, form factors, I will take up in this section; the second, world models, in the next.

While it may appear counterintuitive, we have historically taken it for granted that all humans are of basically equal intelligence. If the spread between the Albert Einsteins and the rest of us seemed wide, it is because we have gotten sensitized to a narrow dynamic range of intelligence. Consider that essentially any two humans sharing a language can attempt to communicate, and, save for specialized topics that have only arisen during the very latest sliver of human history, they can do so effectively and achieve their sought communication goals. Presumably this is because, comparatively speaking, brain sizes and densities are roughly constant across the human population¹⁰. Contrast that with the spread in intelligence between the current best LLMs¹¹ and ChatGPT 3.5, and it becomes clear that two intelligences can be much, much further apart than we have been accustomed to.

The appeal, and challenge, of machine intelligence is that it comes in a range of form factors, from mobile devices to datacenters. This is appealing because it allows intelligence to scale with physical size: larger physical footprint = more GPUs = more model parameters = greater intelligence. It is challenging because our brains have not scaled in this way, and more to the point, because it means the intelligence frontier will get very far from human brains very quickly, and will, at all points in the future, continue to depend on physical size. I.e., there will never¹² be a way to escape the fact that bigger = smarter.

This presents a seemingly fatal challenge to my proposition that humans can reach the intelligence frontier by upgrading their brains, at least unless they are willing to become datacenters. I suspect that some people will want to become datacenters, but I’m getting ahead of myself. What I would like to first do is outline likely aspects of future co-working relationships between machines and enhanced humans, as that will help us reason through the form factor requirements on human-origin brains. (I assume that purely biological humans will be out of the co-working loop.)

The crux of the matter, as I see it, is the size and nature of cognitive tasks that humans will want to engage in. Recalling Amodei’s concept of marginal returns to intelligence, we can suppose that some tasks benefit so little from increased intelligence that humans can already do them as well as they can be done. For such tasks, nothing needs to change, as base human intelligence is good enough.

Next are tasks that require understanding a construction that machines have created or elucidated. Some constructions may be simple enough that even current humans can understand them, if a smart enough person thinks about the construction long enough. This is, incidentally, the category in which current LLM solutions in programming and mathematics reside. LLMs can solve some problems much faster than humans, but their solutions can always be understood by a sufficiently motivated person¹³.

Next are tasks in which human understanding of the construction would be impossible or very slow, say slower than a human lifespan permits. Here, enhanced brains begin to matter. They may speed up cognition to the point that a person can just think through a construction. Or, through external brain modulation, a person may simply gain an understanding, akin to how subconscious thinking can gestate for a time until an idea materializes seemingly unprompted in one’s mind.

Let me expand on this scenario before moving on to the next rung of tasks as it naturally motivates ex vivo computations. If one’s objective is to merely understand complex machine-discovered constructions, then the form factor requirements on human brains may remain quite modest. Massive computations, perhaps encompassing planet-sized datacenters, can return simple, digestible conclusions. The space, energy, and time required by these computations can be decoupled from the human brains that will understand them. Discoveries can be made ex vivo in an ostensibly unconscious manner while understanding happens through conscious reading of their solutions or an unconscious brain download. This exercise is not unfamiliar to us today. For instance, molecular dynamics simulations can be extremely time- and compute-intensive, but their salient conclusions can fit within a few figures.

This scenario highlights communication speed and spatial localization as two complications that will become more serious when we move to the next rung. Depending on how quickly a construction must be understood, the data transfer rates from datacenter to human-origin brain can become a bottleneck. At the point in which lightspeed itself is a limiting factor—like it is in high-frequency trading—ex vivo computation ceases to be an option and the computation must occur in situ. Spatial localization is an extreme version of this where the person and the datacenter are so far apart that they are out of communication range; for instance, a deep space explorer communicating with Earth.

These complications are a fact of life for the next rung, which comprises cognitive tasks that engage in—as opposed to merely understand—computations too large to fit within the normal form factor of human brains. Up till now, we only required understanding. At this rung we move to contributing to and controlling machine computations.

The “easy” solution is to make our brains bigger. This may work for some people but let’s set that aside for a moment. If the communication requirements are tractable, it may be possible to split (“shard”) a computation across in situ and ex vivo substrates, such that it remains a single integrated computation but is performed in a distributed manner. A subset of artificial neurons would sit in a datacenter while another would sit in a human-origin brain. As long as the data transfer and synchronization costs are manageable (akin to current interconnects requirements for model sharding), the computation would proceed as if it had been done exclusively within a datacenter or an abnormally large human-origin brain. How a computation is sharded becomes an important technical consideration, as sharding would need to ensure that the computation is both consciously perceived and controlled by the human portion of the distributed neural network.

Finally, we arrive at the last rung: engagement in computations too large to fit in normally sized brains and that cannot be synchronized fast enough across in situ and ex vivo substrates. At this junction, hard decisions must be made. One’s brain must fundamentally get bigger if one were so keen to engage in this rung of cognitive tasks. At the extremes, additional factors begin to contribute, including heat dissipation and mobility, which is why I said some people may choose to become datacenters. It would be the future day equivalent of (a more extreme version of) a hermit’s life: a physically isolated, immobile existence fully devoted to the life of the mind.

World model of the mind

What I tried to do in the preceding section was to push the question, from a physical standpoint, of how far a human-origin brain can get to the intelligence frontier if that frontier is set by recursively improving machines. The apparent answer is that the capacity to engage with and be at the frontier is by no means guaranteed, as physical considerations on size and speed begin to impinge. A small, mobile brain, even if massively upgraded, will not be able to compete with a planet-sized computer on equal terms. Nonetheless, if sacrifices are made with regards to what we currently perceive to be the human experience, then it is conceivable that, from a physical standpoint, a human-origin brain could once again reach the intelligence frontier in the future.

But these physical considerations ignore a more fundamental question about the future prospects of human-origin intelligence: can world models of purely machine provenance always be made compatible with that of the human mind? To consider this, I have to get back to the business of definitions and explain what I mean by a world model of the human mind.

First, I should say that by “world model” I mean the conventional definition¹⁴, i.e., the latent representation of a (possibly embodied) AI agent that perceives its environment through vision, sound, and other sensory modalities, and develops a model of this perceived world that implicitly encodes physical understanding.

I don’t think it’s controversial to say that human minds have a world model of reality as well. Our brains receive raw sensory data, be it electromagnetic waves, physical pressure, or diffusing organic compounds. These, at the base level, are perceived as sensory qualia, be it vision, sound, or smell. These sensory qualia, along with our internal qualia—i.e., our sense(s) of our mind state—ultimately get integrated into a world model, an artificial construction that our mind creates of what reality is. This construction is not in any fundamental way real. It is a useful summary of reality and its underlying causal mechanisms that evolution bestowed upon us so that we can function well in the world. It also happens to be conscious¹⁵.

It is tempting to say and that’s where the trouble starts, but I think it’s more fundamental than that. My concern is that, at a baser level than the question of consciousness, there is no guarantee that any two world models can meaningfully engage with each other.

There will soon be—in fact there already has been—a Cambrian explosion of intelligence forms and associated world models. LLMs, in all their variants, are obvious examples. So are the AIs behind self-driving cars and game-playing agents like AlphaGo and AlphaStar. Future AIs will reason natively in infinite-dimensional spaces, pilot ships on deep space missions, and perceive genetic sequences the way we perceive the visual world. These will be profoundly alien intelligences, and there is no theorem in cognitive science that says their world models must be compatible with ours. In other words, there may be fundamental limits in thinking-space that supersede physical space.

Then there is consciousness. In the preceding section, we considered human-origin cognitive tasks that involve substantial computational expenditures. One reason to integrate humans in such cognitive loops would be to enable them to consciously experience (and steer) the thinking processes involved in solving problems of substantial magnitude. We do not know if this consciousness overlay is computationally beneficial, neutral, or detrimental. If it is costly, it may place fundamental limits on how large a computation a human-origin mind can feasibly be involved in. We do not, at the moment, have even the beginnings of an answer to this question, with the science of consciousness being as nascent as it is¹⁶.

There may also not be any limits, and we find that it is possible to expand our mind’s world model with new qualia and new states that encompass all the other world models we wish to internalize. It is fun to consider an outlandish projection inspired by the Culture series of books. We know it is possible to broaden the human perceptual range under psychedelics-induced synesthesia. Much of the focus of this essay has been about useful future work, and specifically the exploration of our external and internal worlds, through space travel and (something like) meditation. It seems to me that at least one future we could look forward to is the possibility of turning into, as opposed to flying inside, spaceships of the type enjoyed by the Culture. In such a scenario, we would not merely fly through space but perceptually experience it in an entirely new form factor and world model.

Incidentally, the great AIs of the Culture, the Minds, are always purely machine in origin. There is not the concept of a human-origin mind, one that began with the seed of human consciousness but then expanded in myriad ways. Perhaps there are good reasons for this, a question I will come back to in “My uncertainties”.

Priorities

How quickly will we get to the beginnings of this technology? Given the sci-fi nature of my proposal, it is a little absurd to speak of timelines, let alone ones operating on a post-AGI velocity that will invariably break our current intuitions. But absurdity is my business, and so I will say that a few years seems hopelessly optimistic, even with exponentially improving intelligence. A century, or centuries, seems too far and pointless to try to reason over. Decades, be it two or four, is what I would wager, but that is not really the point of this section.

Rather, the exercise of considering timelines helps reveal two important questions that I expect we will wrestle with collectively as a society. First is prioritization: if we have access to nearly inexhaustible wells of intelligence, what problems should we solve? National funding agencies like the NIH already deal with these types of questions, but the stakes will become much higher, because of the potentially exorbitant material and energy costs needed to compress decades of technological progress into years; the disagreements will also be more contentious, because of the increased stakes and conflicting value systems of different communities and countries.

Second is the question of how priorities are set. In capitalist-leaning democracies, it may come down to who is wealthiest and able to steer the greatest amount of capital towards a particular goal, with competing individuals and organizations setting and pursuing their own agendas. In autocracies, priorities may be set centrally by the government and pursued as national projects. In socialist-leaning democracies, it may become the central part of direct participatory democracy.

Speculating a bit further on this last point, if we were to successfully transition to a post-AGI, post-scarcity society, where universal basic income obviates the need for most people to work, then (inter)national policy may acquire paramount importance in people’s everyday lives, in a way far more salient than today. As things stand, the consequences of political and policy decisions often play out over the course of years and decades, with scientific policy making having even more prolonged timelines. But if the choice between curing cancer and curing Alzheimer’s were to become a real one, exercisable and implementable in a few years, then the consequences of setting scientific priorities will become much more palpable, and interesting, to people. Instead of voting based on vibes, citizens may become so incentivized to influence technological prioritization that they spend substantial fractions of their waking hours considering and researching their options, to both determine their value system and the technological directions most likely to fulfil it. I find it conceivable that “work” in the post-AGI era for people living in participatory democracies would primarily revolve around their exercise of direct democracy.

What may prove to be the limiting factor for human brain enhancement is thus not its inherent scientific difficulty but its prioritization relative to other problems. Aging, for instance, will likely emerge as a top priority¹⁷. So will efficient energy production. Among the countless problems that human societies will want solved, how important is the workaphile’s dilemma in the grand scheme of things?

Civilizational course

While I have focused this essay on AGI’s impact on workaphiles, there is one way in which the post-AGI question extends beyond my tribe, and that is the course of our civilization, i.e., the trajectory, characteristics, and aspirations that our civilization chooses to pursue. This is a distinct question from that of alignment; I am assuming things work out well and there is no life-ending catastrophe on Earth. Civilizational course is about the type of people we want to be—the places and things we explore, the questions we answer, the lives we live. It is a distinction more akin to Buddhism vs. Christianity rather than good vs. bad—different questions; different emphases; different answers.

We are used to thinking about the courses of individual human lives; their achievements and personalities. We are less used to thinking about our civilizational course as a whole, because historically it has moved so slowly, and so organically—usually without any one person influencing its trajectory in any immediately evident way—that we treat our collective future as emergent from the collection of our individual decisions, with no central steering committee.

As we embark on this new epoch however, the calculus may change. On the one hand, timelines are likely to compress, so much so that centuries worth of pre-AGI progress will transpire over the course of individual human lives, making the pace of civilizational progress materially noticeable. On the other hand, the technological directions we choose to prioritize will have far more material and immediate impact than before.

One such choice is whether to cede the intelligence frontier to machines by offloading our thinking to AI, such that we set the highest-level agenda but become ignorant of how that agenda gets translated into concrete actions. Conversely, we may choose to retain the intelligence frontier, by upgrading our brains such that human-origin intelligence not only sets the agenda but can work out its consequences and actively participate in its implementation.

My main point here is not to advocate for one direction over the other, but to highlight that this choice, implicit or explicit, is before us, and that it will impact our civilizational course.

To see this, consider a hypothetical (probably false) choice between a society of immortal humans with 2026 brains vs. a society of mortal humans with vastly upgraded brains. Irrespective of which you prefer, I think it is fair to say that the civilizational trajectories of such two societies will be different, even though it may be hard to work out what those differences will be. And although I present the choice in an unnecessarily dichotomous way, in the extreme case where we would prioritize aging research over all else, we would implicitly be picking the first option.

Reality may also prove to be more dichotomous than we currently anticipate. Both the prolonging of human life and the enhancing of human brains are likely to be enormously complex undertakings, with plausible timelines in the decades even in a post-AGI world. During these decades, humans may offload much of their cognitive workload to machines. Such a prolonged period of attenuated mental activity may leave a permanent imprint. One need only look to the COVID pandemic, in which two years of social isolation led to anti-social behaviors that persist to this day. A humanity that has had its thinking done by machines for decades will not have the same characteristics as the humanity we know today.

Timothy Gowers once again voiced an eloquent version of this concern for mathematicians, which I quote here.

“There is at the moment an extraordinary body of knowledge and expertise that exists not just in the mathematical literature but in the heads of mathematicians all round the world. Imagine if AI didn’t exist and a pandemic broke out that for some reason wiped out all mathematicians and nobody else. All the literature would still be there, but nobody would have the faintest idea what to do with it. To revive a mathematical tradition under those circumstances would be extremely difficult and take decades. Now imagine a slight variant of that, where AI does exist and because of it people are no longer motivated to put in the years of effort it takes to reach the level of expertise that a typical research mathematician has now. After a decade or two, we might arrive at a situation where the mathematical literature has, in some form, been vastly expanded, but there is no corresponding community of human experts who have a shared understanding of parts of it” — Timothy Gowers

The future Gowers fears for mathematical practice may affect humanity writ large.

My uncertainties

Though I have espoused no preference for one civilizational course over another, it is probably no surprise that I reflexively lean towards a society of mortal super intellects over immortal dunces. But I say this, as I say everything else in this essay, with a sense of profound uncertainty. I am deeply, deeply unsure about everything I have said here. I emphasize this uncertainty because we have too many certain voices and it behooves us to come to these problems with humility.

For instance, on the question of civilizational course, while my instinct is to retain as much of its human characteristics as possible, it is also clear that valid arguments can be made for some moderation. For one, human civilization as it exists is far from flawless, and has not been the best steward of the planet or its people. To the extent that we can shift our collective decision-making to be more scientific and freer of ingrained assumptions and biases, we should. For two, human psychology is competitive and in some ways predatory, a function of millennia of evolution. Do we want to endow this psychology with super intelligence? What would human civilization look like if we simply extrapolated its current trajectory but with every individual person multiply more intelligent? A part of me fears that civilizational course, despite preferring it over the alternative.

I also have a nagging sense that what I am proposing is backward-looking and reactionary in outlook. Setting aside its technical complexity, my proposal promises easy answers to difficult and complex questions. It essentially asserts that we can go on as we have been for millennia, engaging in the same types of lives we always had, and retaining stewardship of our planet and civilization—all we need to do, I say, is make ourselves smarter. Something about this idea is too good to be true. We give up almost nothing and gain everything. In the past at least, this has rarely been the case. Great progress came at great cost, and this time the stakes seem infinitely greater.

Writing this, I have a sinking feeling that this essay is so far off base, that it can become a sort of case study in how wrong people were about the future during this era. But not writing it seems cowardly, as it will take many failures and many fools to figure out the next few years, and I would rather that I (and everyone else) take the risk of looking and being foolish than miss the chance of getting it right.

Whither humanity?

I started this blog post by defining “meaning”. I will end it by considering potential evolutions of the word “human”.

Today, “human” is an unambiguous term; I posit that in the future this may cease to be true. Throughout this essay, I used the term “human-origin” to refer to brains that were, at their conception, purely human in the current sense of the word, but that over time get replaced with synthetic parts that upgrade their capabilities but retain salient aspects of their humanity. But what does that mean exactly, and when is the line crossed when a person that was once human no longer is?

Is being human about our physiology; about having been born with two legs and two arms? Clearly many people are not, and we do not consider them the least bit less human for it; quite the opposite. Is it about our senses, our ability to see, to smell, to touch? Again, many people are born or become blind or deaf, and yet they are as human as perfectly abled people are. Is it about our genetics? Perhaps; that is certainly the definition we use today. But consider a biological alien that has all the senses that a human does, that was raised as a human, lived among humans, is culturally human, and has the same cognitive and behavioral range as humans. Can that person not lay some claim to humanness? What about a machine that was made but not born¹⁸, and otherwise perfectly emulates the conscious human experience? Should it be forbidden from claiming humanity as its own?

If I were to proffer an answer, I would say that being human may come to be defined as having a range of internal (not sensory) qualia that is at least a superset of the human ones we are all used to. I.e., to be human is to experience the internal human monologue as we’ve known it, without excluding the capacity for more. Perhaps that expanded capacity will one day be so profound that it renders its human origin inconsequential—at which point, the question may lose its import anyway.

I close with a personal reflection. Throughout much of my life, I often turned over the question of what time period would have been ideal for me to be born in. Far in the future, with routine interplanetary travel? Far in the past, at the dawn of the scientific or industrial revolutions? It is only during the past decade that I stopped wondering, as it dawned on me that I was actually born during the most consequential—and thus to me, ideal—period in our species history; perhaps in our planet’s history. How lucky I am. How lucky we all are. What privilege. What responsibility.

Footnotes

1. Strictly speaking what I said is not true because I used ChatGPT for some quantified-self geekery. Thanks to the venerable LLM I know that this post took me 47 hours and 30 minutes to write. I started it on May 24, 2025 and finished it on Aug 9, 2026. I wrote it over 35 writing sessions split across 13 days. The longest continuous session was 2 hours and 50 minutes and the most I wrote in a day was 6 hours and 30 minutes. ↩︎
2. I do not claim novelty. While I’ve obsessed over this question for a long time, I have by no means pursued other people’s answers with the same vigor. I have read here and there, found some interesting thoughts (which certainly influenced my own), but generally walked away feeling unsatisfied. It is very possible that what I am saying here has been said better by people who think about these questions for a living. ↩︎
3. It has hard to know what the mature form of LLM intelligence will look like, specifically the ways in which it will be (dis)similar to human intelligence, but at least we know that its developmental stages, as they have been chronicled since GPT-1, look very different from the stages of maturation that human children undergo. ↩︎
4. When I first started working on this blog post in May 2025, I wrote “it is also very clear that they [LLMs] have yet to match human intelligence and that they will do so very soon (my guess: 1-2 years) and then quickly exceed it.” It is telling that I am no longer comfortable asserting the first clause of the original sentence. ↩︎
5. In earlier versions of this essay, somewhat tongue-in-cheek, I used “workaholics” as the term, but later decided against it given the loaded connotation which I did not want to imply. ↩︎
6. Some people do play games to de-stress and become better focused at useful work. ↩︎
7. It is often noted with irony that Hardy’s field, number theory, which at the time of his apologia did not have many practical applications, went on to become foundational for cryptography. ↩︎
8. There is an interesting thought experiment in this essay by Kirwin Hampshire, that I quote here, that is effectively the setup for the Glass Bead Game: “What if we told the author that they would never write again. They are forbidden from creating original works to express themselves. However, they are still permitted to comment on writing, interpret it, share their taste. They are still valued for their appraisal, presentation, understanding and appreciation of creative writing. They just can’t write creatively anymore.” — Kirwin Hampshire ↩︎
9. Explain Like I am Human. ↩︎
10. Obviously I’m oversimplifying here and intelligence within the human population is a lot more complex than size/density and is not even correlated with it. This is a statement that is more appropriate across animals than within any single species, but the basic point stands. ↩︎
11. I was tempted to write Fable 5 or Sol, then realized it would get outdated in a few weeks. ↩︎
12. Never is a word that should probably not exist in the English language, as it’s never¹² true. ↩︎
13. This has already been a stretch for things like moves in Go, however. ↩︎
14. At the time of writing there is a trend of renaming any foundation model a “world model”, presumably because it sounds more exciting even if it is nonsense. World models do mean something, however, and that original meaning is what is relevant here. ↩︎
15. For reasons that are not entirely clear. ↩︎
16. I find it curious how questions long perceived as being purely philosophical, and hence somewhat impractical, have taken on exceedingly consequential import at the dawn of the AGI era. Alignment research is full of such questions. ↩︎
17. See Altos Labs. ↩︎
18. I stole the born vs. made distinction from Thinky. ↩︎