The Bounds of Algorithmic Collusion; $Q$-learning, Gradient Learning, and the Folk Theorem

TL;DR

This paper investigates the strategic behavior of learning agents in repeated games, revealing the potential for algorithmic collusion through various learning dynamics and establishing new convergence results for multi-agent Q-learning.

Contribution

It provides a comprehensive analysis of learning dynamics in repeated games, including the first convergence proof for multi-agent Q-learning, and characterizes the set of achievable payoffs, highlighting collusion possibilities.

Findings

Wide range of payoff vectors can be achieved by learning dynamics.

First convergence result for multi-agent Q-learning in repeated games.

Algorithmic collusion can emerge under various learning algorithms.

Abstract

We explore the behaviour emerging from learning agents repeatedly interacting strategically for a wide range of learning dynamics, including $Q$-learning, projected gradient, replicator and log-barrier dynamics. Going beyond the better understood classes of potential games and zero-sum games, we consider the setting of a general repeated game with finite recall under different forms of monitoring. We obtain a Folk Theorem-style result and characterise the set of payoff vectors that can be obtained by these dynamics, discovering a wide range of possibilities for the emergence of algorithmic collusion. Achieving this requires a novel technical approach, which, to the best of our knowledge, yields the first convergence result for multi-agent $Q$-learning algorithms in repeated games.