Anticipatory Fictitious Play

TL;DR

This paper introduces an improved variant of fictitious play that maintains theoretical convergence guarantees while achieving better empirical results, and extends its application to deep multiagent reinforcement learning.

Contribution

It proposes a simple modification of fictitious play that improves empirical performance without sacrificing convergence guarantees and extends its use to deep reinforcement learning.

Findings

Superior empirical performance over traditional fictitious play

Proven optimal convergence rate for certain game classes

Effective extension to deep multiagent reinforcement learning

Abstract

Fictitious play is an algorithm for computing Nash equilibria of matrix games. Recently, machine learning variants of fictitious play have been successfully applied to complicated real-world games. This paper presents a simple modification of fictitious play which is a strict improvement over the original: it has the same theoretical worst-case convergence rate, is equally applicable in a machine learning context, and enjoys superior empirical performance. We conduct an extensive comparison of our algorithm with fictitious play, proving an optimal convergence rate for certain classes of games, demonstrating superior performance numerically across a variety of games, and concluding with experiments that extend these algorithms to the setting of deep multiagent reinforcement learning.