SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

TL;DR

SaFormer introduces a novel conditional sequence modeling approach for offline safe reinforcement learning, enabling constraint satisfaction, adaptability to changing safety requirements, and generalization beyond the training data.

Contribution

The paper proposes SaFormer, a new offline safe RL method using cost tokens and safety verification, improving constraint handling and adaptability over existing approaches.

Findings

Achieves competitive returns with safety constraints

Adapts to new safety costs without retraining

Generalizes to unseen safety constraints

Abstract

Offline safe RL is of great practical relevance for deploying agents in real-world applications. However, acquiring constraint-satisfying policies from the fixed dataset is non-trivial for conventional approaches. Even worse, the learned constraints are stationary and may become invalid when the online safety requirement changes. In this paper, we present a novel offline safe RL approach referred to as SaFormer, which tackles the above issues via conditional sequence modeling. In contrast to existing sequence models, we propose cost-related tokens to restrict the action space and a posterior safety verification to enforce the constraint explicitly. Specifically, SaFormer performs a two-stage auto-regression conditioned by the maximum remaining cost to generate feasible candidates. It then filters out unsafe attempts and executes the optimal action with the highest expected return. Extensive experiments demonstrate the efficacy of SaFormer featuring (1) competitive returns with tightened constraint satisfaction; (2) adaptability to the in-range cost values of the offline data without retraining; (3) generalizability for constraints beyond the current dataset.