Conditional Sequence Modeling for Safe Reinforcement Learning

TL;DR

This paper introduces RCDT, a novel conditional sequence modeling approach for offline safe reinforcement learning that enables zero-shot adaptation to multiple cost thresholds within a single policy, improving return-cost trade-offs.

Contribution

RCDT is the first CSM-based offline safe RL method that supports zero-shot deployment across multiple cost thresholds with an auto-adaptive penalty mechanism.

Findings

RCDT outperforms baseline methods on the DSRL benchmark.

It achieves better return--cost trade-offs.

The approach demonstrates consistent improvements across various scenarios.

Abstract

Offline safe reinforcement learning (RL) aims to learn policies from a fixed dataset while maximizing performance under cumulative cost constraints. In practice, deployment requirements often vary across scenarios, necessitating a single policy that can adapt zero-shot to different cost thresholds. However, most existing offline safe RL methods are trained under a pre-specified threshold, yielding policies with limited generalization and deployment flexibility across cost thresholds. Motivated by recent progress in conditional sequence modeling (CSM), which enables flexible goal-conditioned control by specifying target returns, we propose RCDT, a CSM-based method that supports zero-shot deployment across multiple cost thresholds within a single trained policy. RCDT is the first CSM-based offline safe RL algorithm that integrates a Lagrangian-style cost penalty with an auto-adaptive penalty coefficient. To avoid overly conservative behavior and achieve a more favorable return--cost trade-off, a reward--cost-aware trajectory reweighting mechanism and Q-value regularization are further incorporated. Extensive experiments on the DSRL benchmark demonstrate that RCDT consistently improves return--cost trade-offs over representative baselines, advancing the state-of-the-art in offline safe RL.