Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actions

TL;DR

This paper introduces LSFlow, a novel reinforcement learning policy that uses a latent spherical flow to generate feasible combinatorial actions, improving expressiveness and efficiency in complex decision-making tasks.

Contribution

We propose a solver-induced latent spherical flow policy (LSFlow) that combines generative policy expressiveness with feasibility guarantees for combinatorial RL.

Findings

Outperforms state-of-the-art baselines by 20.6% on average

Learns a stochastic policy in a compact latent space

Uses a smoothed Bellman operator for stable learning

Abstract

Reinforcement learning (RL) with combinatorial action spaces remains challenging because feasible action sets are exponentially large and governed by complex feasibility constraints, making direct policy parameterization impractical. Existing approaches embed task-specific value functions into constrained optimization programs or learn deterministic structured policies, sacrificing generality and policy expressiveness. We propose a solver-induced \emph{latent spherical flow policy} that brings the expressiveness of modern generative policies to combinatorial RL while guaranteeing feasibility by design. Our method, LSFlow, learns a \emph{stochastic} policy in a compact continuous latent space via spherical flow matching, and delegates feasibility to a combinatorial optimization solver that maps each latent sample to a valid structured action. To improve efficiency, we train the value network directly in the latent space, avoiding repeated solver calls during policy optimization. To address the piecewise-constant and discontinuous value landscape induced by solver-based action selection, we introduce a smoothed Bellman operator that yields stable, well-defined learning targets. Empirically, our approach outperforms state-of-the-art baselines by an average of 20.6\% across a range of challenging combinatorial RL tasks.