Exploring the Constraints on Artificial General Intelligence: A Game-Theoretic No-Go Theorem

TL;DR

This paper introduces a game-theoretic framework revealing fundamental constraints on developing superhuman AI, showing that certain assumptions about AI and human interactions cannot all hold simultaneously, guiding policy recommendations.

Contribution

It presents a novel game-theoretic no-go theorem demonstrating the incompatibility of key assumptions in superhuman AI development, informing policy and theoretical understanding.

Findings

Four key assumptions are mutually incompatible when combined.

Relaxing any assumption leads to a consistent framework.

Policy suggestions include controlling data access and research collaboration.

Abstract

The emergence of increasingly sophisticated artificial intelligence (AI) systems have sparked intense debate among researchers, policymakers, and the public due to their potential to surpass human intelligence and capabilities in all domains. In this paper, I propose a game-theoretic framework that captures the strategic interactions between a human agent and a potential superhuman machine agent. I identify four key assumptions: Strategic Unpredictability, Access to Machine's Strategy, Rationality, and Superhuman Machine. The main result of this paper is an impossibility theorem: these four assumptions are inconsistent when taken together, but relaxing any one of them results in a consistent set of assumptions. Two straightforward policy recommendations follow: first, policymakers should control access to specific human data to maintain Strategic Unpredictability; and second, they should grant select AI researchers access to superhuman machine research to ensure Access to Machine's Strategy holds. My analysis contributes to a better understanding of the context that can shape the theoretical development of superhuman AI.