On the Unimportance of Superintelligence

TL;DR

This paper argues that the existential threat from superintelligence is minimal compared to risks from existing peripheral systems like biotechnology and 'savant' software, suggesting resource allocation should focus there.

Contribution

It provides a quantitative analysis showing superintelligence risks are secondary to peripheral system threats, emphasizing the importance of mitigating existing technological risks.

Findings

Superintelligence unconnected to the outside world poses no threat.

Risk from superintelligence is additive and decreases with more humans involved.

Biotechnology and 'savant' software are identified as the primary imminent risks.

Abstract

Humankind faces many existential threats, but has limited resources to mitigate them. Choosing how and when to deploy those resources is, therefore, a fateful decision. Here, I analyze the priority for allocating resources to mitigate the risk of superintelligences. Part I observes that a superintelligence unconnected to the outside world (de-efferented) carries no threat, and that any threat from a harmful superintelligence derives from the peripheral systems to which it is connected, e.g., nuclear weapons, biotechnology, etc. Because existentially-threatening peripheral systems already exist and are controlled by humans, the initial effects of a superintelligence would merely add to the existing human-derived risk. This additive risk can be quantified and, with specific assumptions, is shown to decrease with the square of the number of humans having the capability to collapse civilization. Part II proposes that biotechnology ranks high in risk among peripheral systems because, according to all indications, many humans already have the technological capability to engineer harmful microbes having pandemic spread. Progress in biomedicine and computing will proliferate this threat. ``Savant'' software that is not generally superintelligent will underpin much of this progress, thereby becoming the software responsible for the highest and most imminent existential risk -- ahead of hypothetical risk from superintelligences. The analysis concludes that resources should be preferentially applied to mitigating the risk of peripheral systems and savant software. Concerns about superintelligence are at most secondary, and possibly superfluous.