Escaping the State of Nature: A Hobbesian Approach to Cooperation in Multi-agent Reinforcement Learning

TL;DR

This paper applies Hobbesian political philosophy to multi-agent reinforcement learning, demonstrating that introducing mechanisms inspired by Hobbes can foster cooperation among agents in a novel social dilemma environment.

Contribution

It introduces a new Hobbesian-inspired framework for multi-agent cooperation and demonstrates its effectiveness in a novel social dilemma environment called the Civilization Game.

Findings

Hobbesian mechanisms promote cooperation in multi-agent systems.

Modified Q-Learning agents achieve stable cooperation.

The Civilization Game models the transition from conflict to cooperation.

Abstract

Cooperation is a phenomenon that has been widely studied across many different disciplines. In the field of computer science, the modularity and robustness of multi-agent systems offer significant practical advantages over individual machines. At the same time, agents using standard reinforcement learning algorithms often fail to achieve long-term, cooperative strategies in unstable environments when there are short-term incentives to defect. Political philosophy, on the other hand, studies the evolution of cooperation in humans who face similar incentives to act individualistically, but nevertheless succeed in forming societies. Thomas Hobbes in Leviathan provides the classic analysis of the transition from a pre-social State of Nature, where consistent defection results in a constant state of war, to stable political community through the institution of an absolute Sovereign. This thesis argues that Hobbes's natural and moral philosophy are strikingly applicable to artificially intelligent agents and aims to show that his political solutions are experimentally successful in producing cooperation among modified Q-Learning agents. Cooperative play is achieved in a novel Sequential Social Dilemma called the Civilization Game, which models the State of Nature by introducing the Hobbesian mechanisms of opponent learning awareness and majoritarian voting, leading to the establishment of a Sovereign.