Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

TL;DR

This paper introduces Hybrid Policy Optimization (HPO), a new reinforcement learning method for hybrid discrete-continuous action spaces that combines unbiased mixed gradients and outperforms existing algorithms like PPO in complex control tasks.

Contribution

HPO effectively combines pathwise and score-function gradients for hybrid actions, addressing credit-assignment issues and enabling scalable, unbiased policy optimization in hybrid spaces.

Findings

HPO outperforms PPO on inventory control and switched LQ problems.

Performance gaps increase with higher continuous action dimensions.

The mixed gradient's cross term diminishes near a discrete best response, enabling decentralized updates.

Abstract

We study reinforcement learning in hybrid discrete-continuous action spaces, such as settings where the discrete component selects a regime (or index) and the continuous component optimizes within it -- a structure common in robotics, control, and operations problems. Standard model-free policy gradient methods rely on score-function (SF) estimators and suffer from severe credit-assignment issues in high-dimensional settings, leading to poor gradient quality. On the other hand, differentiable simulation largely sidesteps these issues by backpropagating through a simulator, but the presence of discrete actions or non-smooth dynamics yields biased or uninformative gradients. To address this, we propose Hybrid Policy Optimization (HPO), which backpropagates through the simulator wherever smoothness permits, using a mixed gradient estimator that combines pathwise and SF gradients while maintaining unbiasedness. We also show how problems with action discontinuities can be reformulated in hybrid form, further broadening its applicability. Empirically, HPO substantially outperforms PPO on inventory control and switched linear-quadratic regulator problems, with performance gaps increasing as the continuous action dimension grows. Finally, we characterize the structure of the mixed gradient, showing that its cross term -- which captures how continuous actions influence future discrete decisions -- becomes negligible near a discrete best response, thereby enabling approximate decentralized updates of the continuous and discrete components and reducing variance near optimality. All resources are available at github.com/MatiasAlvo/hybrid-rl.