Beyond Strict Competition: Approximate Convergence of Multi Agent Q-Learning Dynamics

TL;DR

This paper investigates the convergence properties of a smooth variant of multi-agent Q-Learning in near zero-sum games, showing it converges to a neighborhood of equilibrium depending on game proximity and exploration rates.

Contribution

It introduces a convergence analysis for Q-Learning in games close to zero-sum, and provides an efficient method to find the nearest zero-sum game for any network game.

Findings

Q-Learning converges to a neighborhood of equilibrium in near zero-sum games.

The size of the convergence neighborhood depends on the game's distance from zero-sum and exploration rates.

Guarantees hold regardless of whether the dynamics reach an equilibrium or not.

Abstract

The behaviour of multi-agent learning in competitive settings is often considered under the restrictive assumption of a zero-sum game. Only under this strict requirement is the behaviour of learning well understood; beyond this, learning dynamics can often display non-convergent behaviours which prevent fixed-point analysis. Nonetheless, many relevant competitive games do not satisfy the zero-sum assumption. Motivated by this, we study a smooth variant of Q-Learning, a popular reinforcement learning dynamics which balances the agents' tendency to maximise their payoffs with their propensity to explore the state space. We examine this dynamic in games which are `close' to network zero-sum games and find that Q-Learning converges to a neighbourhood around a unique equilibrium. The size of the neighbourhood is determined by the `distance' to the zero-sum game, as well as the exploration rates of the agents. We complement these results by providing a method whereby, given an arbitrary network game, the `nearest' network zero-sum game can be found efficiently. As our experiments show, these guarantees are independent of whether the dynamics ultimately reach an equilibrium, or remain non-convergent.