A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning

TL;DR

This paper introduces a meta-game evaluation framework for deep multiagent reinforcement learning, enabling robust analysis of algorithm interactions and strategic relationships through empirical game sampling and statistical analysis.

Contribution

It proposes a novel meta-game framework that evaluates deep MARL algorithms by sampling empirical games, revealing strategic insights and effects of run-time search.

Findings

Meta-game analysis uncovers strategic relationships among MARL methods.

Run-time search as a meta-strategy improves performance.

Empirical game sampling provides robust evaluation metrics.

Abstract

Evaluating deep multiagent reinforcement learning (MARL) algorithms is complicated by stochasticity in training and sensitivity of agent performance to the behavior of other agents. We propose a meta-game evaluation framework for deep MARL, by framing each MARL algorithm as a meta-strategy, and repeatedly sampling normal-form empirical games over combinations of meta-strategies resulting from different random seeds. Each empirical game captures both self-play and cross-play factors across seeds. These empirical games provide the basis for constructing a sampling distribution, using bootstrapping, over a variety of game analysis statistics. We use this approach to evaluate state-of-the-art deep MARL algorithms on a class of negotiation games. From statistics on individual payoffs, social welfare, and empirical best-response graphs, we uncover strategic relationships among self-play, population-based, model-free, and model-based MARL methods.We also investigate the effect of run-time search as a meta-strategy operator, and find via meta-game analysis that the search version of a meta-strategy generally leads to improved performance.