Deep SOR Minimax Q-learning for Two-player Zero-sum Game

TL;DR

This paper introduces a deep minimax Q-learning algorithm for two-player zero-sum games, extending previous tabular methods to high-dimensional spaces with neural networks, and proves its finite-time convergence.

Contribution

It presents the first deep successive over-relaxation minimax Q-learning algorithm with convergence guarantees for high-dimensional settings.

Findings

The proposed method outperforms existing Q-learning algorithms in numerical experiments.

Ablation studies show the impact of the relaxation parameter on performance.

Abstract

In this work, we consider the problem of a two-player zero-sum game. In the literature, the successive over-relaxation Q-learning algorithm has been developed and implemented, and it is seen to result in a lower contraction factor for the associated Q-Bellman operator resulting in a faster value iteration-based procedure. However, this has been presented only for the tabular case and not for the setting with function approximation that typically caters to real-world high-dimensional state-action spaces. Furthermore, such settings in the case of two-player zero-sum games have not been considered. We thus propose a deep successive over-relaxation minimax Q-learning algorithm that incorporates deep neural networks as function approximators and is suitable for high-dimensional spaces. We prove the finite-time convergence of the proposed algorithm. Through numerical experiments, we show the effectiveness of the proposed method over the existing Q-learning algorithm. Our ablation studies demonstrate the effect of different values of the crucial successive over-relaxation parameter.