Ranking Joint Policies in Dynamic Games using Evolutionary Dynamics

TL;DR

This paper introduces a method to evaluate and rank joint strategies in dynamic games using evolutionary dynamics, specifically employing the $ extalpha$-Rank methodology, to identify stable and long-term effective strategies.

Contribution

It proposes transforming dynamic games into empirical forms based on strategies and applies $ extalpha$-Rank to assess strategy stability and effectiveness over time.

Findings

Identifies strategies resistant to change in stochastic graph coloring

Demonstrates the effectiveness of $ extalpha$-Rank in ranking strategies

Uses DQN-trained policies to generate payoff matrices

Abstract

Game-theoretic solution concepts, such as the Nash equilibrium, have been key to finding stable joint actions in multi-player games. However, it has been shown that the dynamics of agents' interactions, even in simple two-player games with few strategies, are incapable of reaching Nash equilibria, exhibiting complex and unpredictable behavior. Instead, evolutionary approaches can describe the long-term persistence of strategies and filter out transient ones, accounting for the long-term dynamics of agents' interactions. Our goal is to identify agents' joint strategies that result in stable behavior, being resistant to changes, while also accounting for agents' payoffs, in dynamic games. Towards this goal, and building on previous results, this paper proposes transforming dynamic games into their empirical forms by considering agents' strategies instead of agents' actions, and applying the evolutionary methodology $\alpha$-Rank to evaluate and rank strategy profiles according to their long-term dynamics. This methodology not only allows us to identify joint strategies that are strong through agents' long-term interactions, but also provides a descriptive, transparent framework regarding the high ranking of these strategies. Experiments report on agents that aim to collaboratively solve a stochastic version of the graph coloring problem. We consider different styles of play as strategies to define the empirical game, and train policies realizing these strategies, using the DQN algorithm. Then we run simulations to generate the payoff matrix required by $\alpha$-Rank to rank joint strategies.