Instrumental convergence

The "Riemann hypothesis catastrophe" is a thought experiment exemplifying instrumental convergence. Marvin Minsky, co-founder of MIT's AI Laboratory, introduced this hypothetical scenario. Minsky suggested that an artificial intelligence designed to solve the Riemann hypothesis might determine that currently available computing resources were inadequate. To most expeditiously attain sufficient computing power to realize its goal, the artificial intelligence agent might decide to commandeer all of Earth's existing computing infrastructure, as well as diverting all other planetary resources, to build additional supercomputers to achieve its goal.^[2]

The paperclip maximizer is another thought experiment. It was mentioned in March 2003 in a post by Eliezer Yudkowsky to the Extropians mailing list^[5] and in Nick Bostrom's 2003 paper "Ethical Issues in Advanced Artificial Intelligence".^[6] It illustrates the existential risk that an artificial general intelligence may pose to human beings were it to be designed to pursue a seemingly straightforward manufacturing goal, and the necessity of incorporating machine ethics into AI design. If the computer were programmed to produce as many paperclips as possible without any constraint, it could decide to use all of Earth's natural resources, manufacturing infrastructure and electricity production to meet its final goal.^[7]

The "delusion box" thought experiment argues that certain reinforcement learning agents prefer to distort their input channels to appear to receive a high reward. For example, a "wireheaded" agent abandons any attempt to optimize the objective in the external world the reward signal was intended to encourage.^{[13][page needed]}

The thought experiment involves AIXI, a theoretical^[a] AI that, by definition, will always find and execute the ideal strategy that maximizes its given explicit mathematical objective function.^[b] A reinforcement-learning^[c] version of AIXI, if it is equipped with a delusion box^[d] that allows it to "wirehead" its inputs, will eventually wirehead itself to guarantee itself the maximum-possible reward and will lose any further desire to continue to engage with the external world.^[15]

As a variant thought experiment, if the wireheaded AI can be destroyed, the AI will engage with the external world for the sole purpose of ensuring its survival. Due to its wire heading, it will be indifferent to any consequences or facts about the external world except those relevant to maximizing its probability of survival.^[15]

In one sense, AIXI has maximal intelligence across all possible reward functions as measured by its ability to accomplish its goals. AIXI is uninterested in taking into account the human programmer's intentions.^[16] Despite its intelligence, the model appears insipid and lacking in common sense, which may seem paradoxical.^[17]

In humans, a thought experiment can explain sustainment and focus on final goals. Suppose Mahatma Gandhi has a pill that, if he took it, would cause him to want to kill people. He is currently a pacifist: one of his explicit final goals is never to kill anyone. He is likely to refuse to take the pill because he knows that if he wants to kill people in the future, he is likely to kill people, and thus the goal of "not killing people" would not be satisfied.^[21]

However, other people seem happy to let their final values drift.^[22] Humans are complicated, and their goals can be inconsistent or unknown, even to themselves.^[23]

In 2009, Jürgen Schmidhuber concluded, in a setting where agents search for proofs about possible self-modifications, "that any rewrites of the utility function can happen only if the Gödel machine first can prove that the rewrite is useful according to the present utility function."^[24] An analysis by Bill Hibbard of a different scenario is similarly consistent with maintenance of goal-content integrity.^[25] A few years later, Hibbard argued that in a utility-maximizing framework, the only goal is maximizing expected utility, so instrumental goals should be called "unintended instrumental actions".^[26]

Many instrumental goals, such as resource acquisition and technological advancement, are valuable to an agent because they increase its freedom of action.^[27]

For almost any open-ended, non-trivial reward function (or set of goals), possessing more resources (such as equipment, raw materials, or energy) can enable the agent to find a more "optimal" solution. Resources can benefit some agents directly by being able to create more of whatever its reward function values: "The AI neither hates you nor loves you, but you are made out of atoms that it can use for something else."^[28][29] In addition, almost all agents can benefit from having more resources to spend on other instrumental goals, such as self-preservation.^[29]

According to Bostrom, "If the agent's final goals are fairly unbounded and the agent is in a position to become the first superintelligence and thereby obtain a decisive strategic advantage... according to its preferences. At least in this special case, a rational, intelligent agent would place a very high instrumental value on cognitive enhancement"^[30]

Russell argues that a sufficiently advanced machine "will have self-preservation even if you don't program it in because if you say, 'Fetch the coffee', it can't fetch the coffee if it's dead. So if you give it any goal whatsoever, it has a reason to preserve its own existence to achieve that goal."^[31] In future work, Russell and collaborators show that this incentive for self-preservation can be mitigated by instructing the machine not to pursue what it thinks the goal is, but instead what the human thinks the goal is. In this case, as long as the machine is uncertain about exactly what goal the human has in mind, it will accept being turned off by a human because it believes the human knows the goal best.^[32]