Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spaces

TL;DR

This paper introduces Distance-Guided Reinforcement Learning (DGRL), a novel method that efficiently handles extremely large discrete action spaces by combining neighborhood sampling and distance-based updates, improving performance and convergence.

Contribution

DGRL is the first approach to enable scalable RL in massive discrete action spaces using stochastic volumetric exploration and stable policy regression.

Findings

DGRL achieves up to 66% performance improvement over state-of-the-art methods.

DGRL guarantees local value improvement on structured tasks.

DGRL improves convergence speed and reduces computational complexity.

Abstract

Reinforcement Learning (RL) is increasingly applied to large-scale decision-making problems like logistics, scheduling, and recommender systems, but existing algorithms struggle with the curse of dimensionality in such large discrete action spaces. We propose Distance-Guided Reinforcement Learning (DGRL), combining Sampled Dynamic Neighborhoods and Distance-Based Updates to enable efficient RL in problems with up to $10^{20}$ actions. Unlike prior methods, DGRL performs stochastic volumetric exploration and transforms policy optimization into a stable regression task, decoupling gradient variance from action space cardinality. On structured tasks, DGRL provably guarantees local value improvement. DGRL naturally generalizes to hybrid continuous-discrete action spaces. We demonstrate performance improvements of up to 66% against state-of-the-art benchmarks across regularly and irregularly structured environments, while simultaneously improving convergence speed and computational complexity.