Learning Markov Games with Adversarial Opponents: Efficient Algorithms and Fundamental Limits

TL;DR

This paper investigates no-regret learning in Markov games with adversarial opponents, providing efficient algorithms with regret bounds under certain conditions and establishing fundamental limits through hardness results.

Contribution

It introduces new algorithms with regret guarantees for Markov games with revealed opponent policies and proves hardness results when policies are hidden, highlighting the limits of learnability.

Findings

Algorithms achieve $ oot{K}$-regret with known opponent policies under certain conditions

Exponential lower bounds when opponent policies are unknown and conditions are not met

Statistical hardness results surpassing previous computational hardness findings

Abstract

An ideal strategy in zero-sum games should not only grant the player an average reward no less than the value of Nash equilibrium, but also exploit the (adaptive) opponents when they are suboptimal. While most existing works in Markov games focus exclusively on the former objective, it remains open whether we can achieve both objectives simultaneously. To address this problem, this work studies no-regret learning in Markov games with adversarial opponents when competing against the best fixed policy in hindsight. Along this direction, we present a new complete set of positive and negative results: When the policies of the opponents are revealed at the end of each episode, we propose new efficient algorithms achieving $\sqrt{K}$-regret bounds when either (1) the baseline policy class is small or (2) the opponent's policy class is small. This is complemented with an exponential lower bound when neither conditions are true. When the policies of the opponents are not revealed, we prove a statistical hardness result even in the most favorable scenario when both above conditions are true. Our hardness result is much stronger than the existing hardness results which either only involve computational hardness, or require further restrictions on the algorithms.