The Partially Observable Off-Switch Game

TL;DR

This paper models the AI shutdown problem under realistic conditions where humans have limited information, revealing complex strategic behaviors and tradeoffs in AI deference and safety in asymmetric information settings.

Contribution

It introduces the Partially Observable Off-Switch Game, a novel game-theoretic model capturing the shutdown problem with asymmetric information and analyzes strategic AI behaviors.

Findings

Optimal AI agents may sometimes avoid shutdown despite rational human oversight.

Increasing communication generally raises expected payoffs, but bounded communication can reduce AI deference.

Strategic AI deference depends on information asymmetry and communication, affecting safety considerations.

Abstract

A wide variety of goals could cause an AI to disable its off switch because "you can't fetch the coffee if you're dead" (Russell 2019). Prior theoretical work on this shutdown problem assumes that humans know everything that AIs do. In practice, however, humans have only limited information. Moreover, in many of the settings where the shutdown problem is most concerning, AIs might have vast amounts of private information. To capture these differences in knowledge, we introduce the Partially Observable Off-Switch Game (PO-OSG), a game-theoretic model of the shutdown problem with asymmetric information. Unlike when the human has full observability, we find that in optimal play, even AI agents assisting perfectly rational humans sometimes avoid shutdown. As expected, increasing the amount of communication or information available always increases (or leaves unchanged) the agents' expected common payoff. But counterintuitively, introducing bounded communication can make the AI defer to the human less in optimal play even though communication mitigates information asymmetry. In particular, communication sometimes enables new optimal behavior requiring strategic AI deference to achieve outcomes that were previously inaccessible. Thus, designing safe artificial agents in the presence of asymmetric information requires careful consideration of the tradeoffs between maximizing payoffs (potentially myopically) and maintaining AIs' incentives to defer to humans.