Constant-Memory Strategies in Stochastic Games: Best Responses and Equilibria

TL;DR

This paper explores constant-memory strategies in stochastic games, analyzing their best responses and equilibria, and verifies theoretical insights through experiments on classic multi-agent decision-making testbeds.

Contribution

It provides new theoretical results on best responses and Nash equilibria for constant-memory strategies and discusses their computational hardness, supported by empirical experiments.

Findings

Best response computations are computationally hard.

Nash equilibria for constant-memory strategies are characterized.

Experimental results validate theoretical insights on testbeds.

Abstract

Stochastic games have become a prevalent framework for studying long-term multi-agent interactions, especially in the context of multi-agent reinforcement learning. In this work, we comprehensively investigate the concept of constant-memory strategies in stochastic games. We first establish some results on best responses and Nash equilibria for behavioral constant-memory strategies, followed by a discussion on the computational hardness of best responding to mixed constant-memory strategies. Those theoretic insights are later verified on several sequential decision-making testbeds, including the $\textit{Iterated Prisoner's Dilemma}$, the $\textit{Iterated Traveler's Dilemma}$, and the $\textit{Pursuit}$ domain. This work aims to enhance the understanding of theoretical issues in single-agent planning under multi-agent systems, and uncover the connection between decision models in single-agent and multi-agent contexts. The code is available at $\texttt{https://github.com/Fernadoo/Const-Mem.}$