PAC Reinforcement Learning Algorithm for General-Sum Markov Games

TL;DR

This paper introduces a PAC reinforcement learning framework for general-sum Markov games, extending Nash Q-learning with delayed Q-learning to ensure provable guarantees and robustness.

Contribution

It develops a new PAC MARL algorithm for general-sum Markov games based on an extension of Nash Q-learning, with a framework to verify PAC properties of MARL algorithms.

Findings

The proposed algorithm is provably PAC.

Numerical results show strong performance and robustness.

Framework allows checking PAC properties of MARL algorithms.

Abstract

This paper presents a theoretical framework for probably approximately correct (PAC) multi-agent reinforcement learning (MARL) algorithms for Markov games. The paper offers an extension to the well-known Nash Q-learning algorithm, using the idea of delayed Q-learning, in order to build a new PAC MARL algorithm for general-sum Markov games. In addition to guiding the design of a provably PAC MARL algorithm, the framework enables checking whether an arbitrary MARL algorithm is PAC. Comparative numerical results demonstrate performance and robustness.