The Expected Value of Artificial General Intelligence

If you are intellectually honest, you should be taking the risk articulated in Eliezer Yudkowsky's Time op-ed seriously. Should humanity collectively and indefinitely halt the development of AGI?

The expected value [1] of AGI looks something like

EV = (p x ∞) + ((1 - p) x -∞)

where (p x ∞) represents the probability of alignment producing an infinitely good outcome and ((1 - p) x -∞) represents the probability of an unaligned super-intelligence producing an infinitely bad outcome.

Why ∞ and -∞ as our respective outcomes? Eliezer argues, and I agree, that the worst case scenario resulting from nonalignment is the permanent extinction of humanity (and all other life on earth). Let it sink in for a moment that permanent extinction eliminates all potential states of human existence within the indefinite future. Is there a more perfect symbolic representation of this immeasurability (not to mention the very destruction of measurability itself) than -∞? Thus, the prevention of -∞ is ∞, as it maintains the potential for infinite non-negative future states.

By definition, ∞ allows for states that lead to -∞ by other means, such as self-annihilation by nuclear weapons. I agree with Eliezer that humanity should be agnostic as to how we prevent self-annihilation. What then is the likelihood that this -∞ is only avoidable if we develop AGI?

If this likelihood is greater than (1 - p), our best bet is to proceed. In other words, we should not halt the development of Artificial General Intelligence if it is more likely to save us from self-extinction by any other means than wipe us out itself. Either way we must face down the infinite and oblivion and make a decision.

Notes

• Expected value is typically used by investors to calculate uncertainty-weighted risk vs. reward for a potential investment.