Offline Safe Policy Optimization From Heterogeneous Feedback

TL;DR

This paper introduces PreSa, a novel offline safe policy optimization framework that directly learns safe, reward-maximizing policies from human preferences and safety labels without explicit reward or cost models, improving safety and performance.

Contribution

The paper proposes a direct policy learning framework using preference and safety data, avoiding explicit reward and cost models, and employs a Lagrangian approach for safe policy optimization.

Findings

Successfully learns safe policies with high rewards in continuous control tasks.

Outperforms state-of-the-art baselines and offline safe RL methods.

Effective with both synthetic and real human feedback.

Abstract

Offline Preference-based Reinforcement Learning (PbRL) learns rewards and policies aligned with human preferences without the need for extensive reward engineering and direct interaction with human annotators. However, ensuring safety remains a critical challenge across many domains and tasks. Previous works on safe RL from human feedback (RLHF) first learn reward and cost models from offline data, then use constrained RL to optimize a safe policy. While such an approach works in the contextual bandits settings (LLMs), in long horizon continuous control tasks, errors in rewards and costs accumulate, leading to impairment in performance when used with constrained RL methods. To address these challenges, (a) instead of indirectly learning policies (from rewards and costs), we introduce a framework that learns a policy directly based on pairwise preferences regarding the agent's behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments; (b) we propose \textsc{PreSa} (Preference and Safety Alignment), a method that combines preference learning module with safety alignment in a constrained optimization problem. This optimization problem is solved within a Lagrangian paradigm that directly learns reward-maximizing safe policy \textit{without explicitly learning reward and cost models}, avoiding the need for constrained RL; (c) we evaluate our approach on continuous control tasks with both synthetic and real human feedback. Empirically, our method successfully learns safe policies with high rewards, outperforming state-of-the-art baselines, and offline safe RL approaches with ground-truth reward and cost.