Structured Reinforcement Learning for Combinatorial Decision-Making

TL;DR

This paper introduces Structured Reinforcement Learning (SRL), a new approach embedding combinatorial optimization into RL actors, enabling better handling of complex decision spaces and outperforming traditional methods in various uncertain environments.

Contribution

The paper presents SRL, a novel actor-critic framework with combinatorial optimization layers, offering end-to-end training and a geometric interpretation as a primal-dual algorithm.

Findings

SRL matches or exceeds unstructured RL and imitation learning on static tasks.

SRL improves performance by up to 92% on dynamic problems.

SRL demonstrates enhanced stability and faster convergence.

Abstract

Reinforcement learning (RL) is increasingly applied to real-world problems involving complex and structured decisions, such as routing, scheduling, and assortment planning. These settings challenge standard RL algorithms, which struggle to scale, generalize, and exploit structure in the presence of combinatorial action spaces. We propose Structured Reinforcement Learning (SRL), a novel actor-critic paradigm that embeds combinatorial optimization-layers into the actor neural network. We enable end-to-end learning of the actor via Fenchel-Young losses and provide a geometric interpretation of SRL as a primal-dual algorithm in the dual of the moment polytope. Across six environments with exogenous and endogenous uncertainty, SRL matches or surpasses the performance of unstructured RL and imitation learning on static tasks and improves over these baselines by up to 92% on dynamic problems, with improved stability and convergence speed.