Co-Learning Empirical Games and World Models

TL;DR

This paper introduces Dyna-PSRO, a novel algorithm that co-learns empirical games and world models to improve strategic decision-making, achieving lower regret and requiring fewer experiences in complex game environments.

Contribution

The paper proposes a new co-learning algorithm, Dyna-PSRO, that integrates empirical game modeling with world dynamics to enhance strategic planning and efficiency.

Findings

Dyna-PSRO achieves lower regret than PSRO in experiments.

Dyna-PSRO requires fewer experiences to learn effective strategies.

Co-learning empirical games and world models improves decision-making in complex games.

Abstract

Game-based decision-making involves reasoning over both world dynamics and strategic interactions among the agents. Typically, empirical models capturing these respective aspects are learned and used separately. We investigate the potential gain from co-learning these elements: a world model for dynamics and an empirical game for strategic interactions. Empirical games drive world models toward a broader consideration of possible game dynamics induced by a diversity of strategy profiles. Conversely, world models guide empirical games to efficiently discover new strategies through planning. We demonstrate these benefits first independently, then in combination as realized by a new algorithm, Dyna-PSRO, that co-learns an empirical game and a world model. When compared to PSRO -- a baseline empirical-game building algorithm, Dyna-PSRO is found to compute lower regret solutions on partially observable general-sum games. In our experiments, Dyna-PSRO also requires substantially fewer experiences than PSRO, a key algorithmic advantage for settings where collecting player-game interaction data is a cost-limiting factor.