If you are intellectually honest, you should be taking the risk articulated in Eliezer Yudkowsky’s Time op-ed seriously. Should humanity collectively and indefinitely halt the development of AGI?
The expected value [1] of AGI looks something like
where represents the probability of alignment producing an infinitely good outcome and represents the probability of an unaligned super-intelligence producing an infinitely bad outcome.
Why and as our respective outcomes? Eliezer argues, and I agree, that the worst case scenario resulting from nonalignment is the permanent extinction of humanity (and all other life on earth). Let it sink in for a moment that permanent extinction eliminates all potential states of human existence within the indefinite future. Is there a more perfect symbolic representation of this immeasurability (not to mention the very destruction of measurability itself) than ? Thus, the prevention of is , as it maintains the potential for infinite non-negative future states.
By definition, allows for states that lead to by other means, such as self-annihilation by nuclear weapons. I agree with Eliezer that humanity should be agnostic as to how we prevent self-annihilation. What then is the likelihood that this is only avoidable if we develop AGI?
If this likelihood is greater than , our best bet is to proceed. In other words, we should not halt the development of Artificial General Intelligence if it is more likely to save us from self-extinction by any other means than wipe us out itself. Either way we must face down the infinite and oblivion and make a decision.
Notes
[1] Expected value is typically used by investors to calculate uncertainty-weighted risk vs. reward for a potential investment.