DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play

TL;DR

DiffFP introduces a diffusion-based fictitious play framework that enables learning robust, adaptive strategies in continuous multi-agent environments, achieving faster convergence and higher success rates than traditional reinforcement learning methods.

Contribution

This paper presents a novel diffusion-based fictitious play approach that estimates best responses to unseen opponents, improving convergence speed and robustness in continuous multi-agent games.

Findings

Converges to $$-Nash equilibria in continuous zero-sum games.

Achieves up to 3x faster convergence than baseline RL methods.

Attains 30x higher success rates against diverse opponents.

Abstract

Self-play reinforcement learning has demonstrated significant success in learning complex strategic and interactive behaviors in competitive multi-agent games. However, achieving such behaviors in continuous decision spaces remains challenging. Ensuring adaptability and generalization in self-play settings is critical for achieving competitive performance in dynamic multi-agent environments. These challenges often cause methods to converge slowly or fail to converge at all to a Nash equilibrium, making agents vulnerable to strategic exploitation by unseen opponents. To address these challenges, we propose DiffFP, a fictitious play (FP) framework that estimates the best response to unseen opponents while learning a robust and multimodal behavioral policy. Specifically, we approximate the best response using a diffusion policy that leverages generative modeling to learn adaptive and diverse strategies. Through empirical evaluation, we demonstrate that the proposed FP framework converges towards $\epsilon$-Nash equilibria in continuous- space zero-sum games. We validate our method on complex multi-agent environments, including racing and multi-particle zero-sum games. Simulation results show that the learned policies are robust against diverse opponents and outperform baseline reinforcement learning policies. Our approach achieves up to 3$\times$ faster convergence and 30$\times$ higher success rates on average against RL-based baselines, demonstrating its robustness to opponent strategies and stability across training iterations