SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

TL;DR

SafeDPO introduces a simple, theoretically grounded method for safety alignment in large language models that improves safety without sacrificing helpfulness, using preference data and minimal modifications.

Contribution

The paper presents SafeDPO, a novel, lightweight safety optimization approach that directly optimizes a safety-constrained objective without auxiliary models or complex pipelines.

Findings

SafeDPO achieves better safety-helpfulness trade-offs than existing methods.

It scales effectively to LLMs with up to 13 billion parameters.

The hyperparameter enhances safety while maintaining theoretical optimality.

Abstract

As Large Language Models (LLMs) are increasingly deployed in real-world applications, balancing helpfulness and safety has become a central challenge. A natural approach is to incorporate safety constraints into Reinforcement Learning from Human Feedback (RLHF), where recent studies have shown promising progress. However, these methods often rely on auxiliary networks or multi-stage pipelines, thereby increasing complexity. In this work, we revisit the original safety alignment objective and show that, under mild assumptions, it admits a closed-form optimal policy. We further derive a provably equivalent and tractable objective, enabling direct optimization. Building on this insight, we propose SafeDPO, a lightweight method that preserves the optimal solution of the underlying safety-constrained objective while requiring only one additional hyperparameter and minimal modifications to existing preference-based training methods. SafeDPO eliminates the need for reward models, cost models, and online sampling, relying only on preference data and safety indicators. Despite its simplicity, SafeDPO achieves competitive safety-helpfulness trade-offs compared to existing safety alignment methods. Experiments on the PKU-SafeRLHF-30K benchmark demonstrate that SafeDPO substantially improves safety while maintaining competitive helpfulness. Ablation studies further show that the additional hyperparameter provides a flexible mechanism to enhance safety while preserving the theoretical optimum, and confirm that SafeDPO scales reliably to LLMs with up to 13B parameters. Overall, our results highlight that a simple, theory-driven objective can provide a lightweight yet effective solution for safety alignment in practice.