Mutation-Bias Learning in Games

TL;DR

This paper introduces two simple multi-agent reinforcement learning algorithms inspired by evolutionary game theory, with proven convergence properties and empirical comparisons to existing methods, highlighting their robustness in high-dimensional settings.

Contribution

It presents novel simple algorithms with theoretical convergence guarantees and demonstrates their effectiveness compared to Q-learning variants in multi-agent environments.

Findings

Proven convergence conditions for the simpler variant.

Experimental results showing robustness in high-dimensional settings.

Comparison with WoLF-PHC and frequency-adjusted Q-learning.

Abstract

We present two variants of a multi-agent reinforcement learning algorithm based on evolutionary game theoretic considerations. The intentional simplicity of one variant enables us to prove results on its relationship to a system of ordinary differential equations of replicator-mutator dynamics type, allowing us to present proofs on the algorithm's convergence conditions in various settings via its ODE counterpart. The more complicated variant enables comparisons to Q-learning based algorithms. We compare both variants experimentally to WoLF-PHC and frequency-adjusted Q-learning on a range of settings, illustrating cases of increasing dimensionality where our variants preserve convergence in contrast to more complicated algorithms. The availability of analytic results provides a degree of transferability of results as compared to purely empirical case studies, illustrating the general utility of a dynamical systems perspective on multi-agent reinforcement learning when addressing questions of convergence and reliable generalisation.