If you are intellectually honest, you should be taking the risk articulated in Eliezer Yudkowsky’s Time op-ed seriously. Should humanity collectively and indefinitely halt the development of AGI?
The expected value  of AGI looks something like
where represents the probability of alignment producing an infinitely good outcome and represents the probability of an unaligned super-intelligence producing an infinitely bad outcome. Why and as our respective outcomes? Eliezer argues, and I agree, that the worst case scenario resulting from nonalignment is the permanent extinction of humanity (and all other life on earth). Let it sink in for a moment that permanent extinction eliminates all potential states of human existence within the indefinite future. Is there a more perfect symbolic representation of this immeasurability (not to mention the very destruction of measurability itself) than ? Thus, the prevention of is , as it maintains the potential for infinite non-negative future states. By definition, allows for states that lead to by other means, such as self-annihilation by nuclear weapons. I agree with Eliezer that humanity should be agnostic as to how we prevent self-annihilation. What then is the likelihood that this is only avoidable if we develop AGI? If this likelihood is greater than , our best bet is to proceed. In other words, we should not halt the development of Artificial General Intelligence if it is more likely to save us from self-extinction by any other means than wipe us out itself. Either way we must face down the infinite and oblivion and make a decision.
 Expected value is typically used by investors to calculate uncertainty-weighted risk vs. reward for a potential investment.