The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes

TL;DR

This paper introduces the Intelligent Disobedience Game, a formal framework combining game theory and Markov Decision Processes to model and analyze safe disobedience in AI systems interacting with humans.

Contribution

It formalizes the concept of intelligent disobedience using Stackelberg games and MDPs, providing a foundation for developing and studying safe AI disobedience strategies.

Findings

Identification of 'safety traps' in disobedience scenarios

Development of a mathematical framework for safe AI disobedience

Creation of a computational testbed for reinforcement learning agents

Abstract

In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as ``safety traps,'' where the system indefinitely avoids harm but fails to achieve the human's goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents.