Instrumental convergence

Instrumental convergence is the hypothetical tendency for most sufficiently intelligent, goal directed beings (human and non-human) to pursue similar sub-goals, even if their ultimate goals are quite different.^[1] More precisely, agents (beings with agency) may pursue instrumental goals—goals which are made in pursuit of some particular end, but are not the end goals themselves—without ceasing, provided that their ultimate (intrinsic) goals may never be fully satisfied.

Instrumental convergence posits that an intelligent agent with unbounded but harmless goals can act in surprisingly harmful ways. For example, a computer with the sole, unconstrained purpose of solving a complex mathematics problem like the Riemann hypothesis could attempt to turn the entire Earth into one giant computer to increase its computational power so that it can succeed in its calculations.^[2]

Proposed basic AI drives include utility function or goal-content integrity, self-protection, freedom from interference, self-improvement, and non-satiable acquisition of additional resources.