Explore Reinforced: Equilibrium Approximation with Reinforcement Learning

TL;DR

This paper introduces a novel algorithm combining reinforcement learning and game theory to improve equilibrium approximation in large stochastic environments, demonstrating enhanced performance in cybersecurity and bandit problems.

Contribution

The paper presents Exp3-IXrl, a new method that separates action selection from equilibrium computation, expanding the applicability of equilibrium algorithms to complex environments.

Findings

Improved equilibrium approximation in large stochastic games.

Enhanced performance in cybersecurity network simulations.

Effective in classical multi-armed bandit problems.

Abstract

Current approximate Coarse Correlated Equilibria (CCE) algorithms struggle with equilibrium approximation for games in large stochastic environments but are theoretically guaranteed to converge to a strong solution concept. In contrast, modern Reinforcement Learning (RL) algorithms provide faster training yet yield weaker solutions. We introduce Exp3-IXrl - a blend of RL and game-theoretic approach, separating the RL agent's action selection from the equilibrium computation while preserving the integrity of the learning process. We demonstrate that our algorithm expands the application of equilibrium approximation algorithms to new environments. Specifically, we show the improved performance in a complex and adversarial cybersecurity network environment - the Cyber Operations Research Gym - and in the classical multi-armed bandit settings.