Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibrium

TL;DR

This paper introduces a novel online learning algorithm for two-player zero-sum Markov Games with nonlinear function approximation in RKHS, achieving sublinear regret and handling high-dimensional exploration.

Contribution

It develops a new algorithm using confidence bounds and optimism principles for Nash equilibrium learning in high-dimensional spaces with theoretical guarantees.

Findings

Achieves $O(\sqrt{T})$ regret with polynomial complexity

Extends to Bernstein-type bonuses for tighter regret bounds

Handles model misspecification with neural function approximation

Abstract

We consider learning Nash equilibria in two-player zero-sum Markov Games with nonlinear function approximation, where the action-value function is approximated by a function in a Reproducing Kernel Hilbert Space (RKHS). The key challenge is how to do exploration in the high-dimensional function space. We propose a novel online learning algorithm to find a Nash equilibrium by minimizing the duality gap. At the core of our algorithms are upper and lower confidence bounds that are derived based on the principle of optimism in the face of uncertainty. We prove that our algorithm is able to attain an $O(\sqrt{T})$ regret with polynomial computational complexity, under very mild assumptions on the reward function and the underlying dynamic of the Markov Games. We also propose several extensions of our algorithm, including an algorithm with Bernstein-type bonus that can achieve a tighter regret bound, and another algorithm for model misspecification that can be applied to neural function approximation.