If you are intellectually honest, you should be taking the risk articulated in Eliezer Yudkowsky’s Time op-ed seriously. Should humanity collectively and indefinitely halt the development of AGI?

The expected value [1] of AGI looks something like

$EV = (p * \infty) + ((1-p) * -\infty)$

where $(p * \infty)$ represents the probability of alignment producing an infinitely good outcome and $((1-p)*-\infty)$ represents the probability of an unaligned super-intelligence producing an infinitely bad outcome.

Why $\infty$ and $-\infty$ as our respective outcomes? Eliezer argues, and I agree, that the worst case scenario resulting from nonalignment is the permanent extinction of humanity (and all other life on earth). Let it sink in for a moment that permanent extinction eliminates all potential states of human existence within the indefinite future. Is there a more perfect symbolic representation of this immeasurability (not to mention the very destruction of measurability itself) than $-\infty$? Thus, the prevention of $- \infty$ is $\infty$, as it maintains the potential for infinite non-negative future states.

By definition, $\infty$ allows for states that lead to $-\infty$ by other means, such as self-annihilation by nuclear weapons. I agree with Eliezer that humanity should be agnostic as to how we prevent self-annihilation. What then is the likelihood that this $-\infty$ is only avoidable if we develop AGI?

If this likelihood is greater than $(1-p)$, our best bet is to proceed. In other words, we should not halt the development of Artificial General Intelligence if it is more likely to save us from self-extinction by any other means than wipe us out itself. Either way we must face down the infinite and oblivion and make a decision.

Notes

[1] Expected value is typically used by investors to calculate uncertainty-weighted risk vs. reward for a potential investment.