Can Reinforcement Learning Solve Asymmetric Combinatorial-Continuous Zero-Sum Games?

TL;DR

This paper introduces a new class of asymmetric zero-sum games called ACCES, proves the existence of Nash equilibrium, and develops novel algorithms combining double oracle methods with reinforcement learning to solve these complex games effectively.

Contribution

The paper defines ACCES games, proves NE existence, and proposes CCDO and CCDORL algorithms with convergence analysis for solving these games in practical combinatorial optimization contexts.

Findings

Proved NE existence for ACCES games.

Developed the CCDO algorithm with proven convergence.

Demonstrated empirical effectiveness of CCDORL on COP instances.

Abstract

There have been extensive studies on learning in zero-sum games, focusing on the analysis of the existence and algorithmic convergence of Nash equilibrium (NE). Existing studies mainly focus on symmetric games where the strategy spaces of the players are of the same type and size. For the few studies that do consider asymmetric games, they are mostly restricted to matrix games. In this paper, we define and study a new practical class of asymmetric games called two-player Asymmetric Combinatorial-Continuous zEro-Sum (ACCES) games, featuring a combinatorial action space for one player and an infinite compact space for the other. Such ACCES games have broad implications in the real world, particularly in combinatorial optimization problems (COPs) where one player optimizes a solution in a combinatorial space, and the opponent plays against it in an infinite (continuous) compact space (e.g., a nature player deciding epistemic parameters of the environmental model). Our first key contribution is to prove the existence of NE for two-player ACCES games, using the idea of essentially finite game approximation. Building on the theoretical insights and double oracle (DO)-based solutions to complex zero-sum games, our second contribution is to design the novel algorithm, Combinatorial Continuous DO (CCDO), to solve ACCES games, and prove the convergence of the proposed algorithm. Considering the NP-hardness of most COPs and recent advancements in reinforcement learning (RL)-based solutions to COPs, our third contribution is to propose a practical algorithm to solve NE in the real world, CCDORL (based on CCDO), and provide the novel convergence analysis in the ACCES game. Experimental results across diverse instances of COPs demonstrate the empirical effectiveness of our algorithms. The code of this work is available at https://github.com/wmd3i/CCDO-RL.