Will Holley

The Expected Value of Artificial General Intelligence

Responding to Eliezer’s Time op-ed

If you are intellectually honest, you should be taking the risk articulated in Eliezer Yudkowsky’s Time op-ed seriously. Should humanity collectively and indefinitely halt the development of AGI?
The expected value [1] of AGI looks something like
EV=(p)+((1p))EV = (p * \infty) + ((1-p) * -\infty)
where (p)(p * \infty) represents the probability of alignment producing an infinitely good outcome and ((1p))((1-p)*-\infty) represents the probability of an unaligned super-intelligence producing an infinitely bad outcome.
Why \infty and -\infty as our respective outcomes? Eliezer argues, and I agree, that the worst case scenario resulting from nonalignment is the permanent extinction of humanity (and all other life on earth). Let it sink in for a moment that permanent extinction eliminates all potential states of human existence within the indefinite future. Is there a more perfect symbolic representation of this immeasurability (not to mention the very destruction of measurability itself) than -\infty? Thus, the prevention of - \infty is \infty, as it maintains the potential for infinite non-negative future states.
By definition, \infty allows for states that lead to -\infty by other means, such as self-annihilation by nuclear weapons. I agree with Eliezer that humanity should be agnostic as to how we prevent self-annihilation. What then is the likelihood that this -\infty is only avoidable if we develop AGI?
If this likelihood is greater than (1p)(1-p), our best bet is to proceed. In other words, we should not halt the development of Artificial General Intelligence if it is more likely to save us from self-extinction by any other means than wipe us out itself. Either way we must face down the infinite and oblivion and make a decision.
[1] Expected value is typically used by investors to calculate uncertainty-weighted risk vs. reward for a potential investment.