Implicit Safety Alignment from Crowd Preferences

TL;DR

This paper introduces a hierarchical framework that learns shared safety criteria from crowd preferences to improve safety in reinforcement learning tasks without explicit safety rewards.

Contribution

It proposes Safe Crowd Preference-based RL, which extracts safety-aligned skills from crowd data and composes them to enhance safety in downstream tasks.

Findings

Reduces safety costs significantly in experiments.

Achieves task performance comparable to methods with ground-truth safety signals.

Identifies limitations of reward combination methods for safety objectives.

Abstract

Reinforcement Learning from Human Feedback (RLHF) can reveal implicit objectives such as safety considerations that go beyond task completion. In this work, we focus on the common safety criteria embedded in crowd preference datasets, where different users may express distinct preferences or objectives, yet follow similar safety principles. Our aim is to discover shared safety criteria from crowd preferences and then transfer them to downstream RL tasks to regularize agent behavior and enforce safety. We first show that direct reward combination-optimizing a preference-learned reward model together with downstream task rewards-has inherent limitations. Motivated by this, we propose Safe Crowd Preference-based RL, a hierarchical framework that extracts safety-aligned skills from crowd preferences and composes them via a high-level policy to safely solve downstream tasks. Experiments across safe RL environments and a preliminary LLM-style task with diverse user goals and shared safety constraints demonstrate that our approach substantially lowers safety costs without access to explicit safety rewards, while achieving task performance comparable to oracle methods trained with ground-truth safety signals.