Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment

TL;DR

This paper introduces Alignment-Weighted DPO, a novel fine-tuning method that enhances large language model safety by focusing on reasoning segments, thereby improving robustness against jailbreak attacks while preserving utility.

Contribution

It proposes a new alignment training approach that assigns differential weights to reasoning and answer segments, improving safety and robustness of LLMs against deceptive prompts.

Findings

Enhanced safety robustness against jailbreak strategies.

Maintained high utility performance of models.

Outperformed standard fine-tuning baselines.

Abstract

Recent advances in alignment techniques such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO) have improved the safety of large language models (LLMs). However, these LLMs remain vulnerable to jailbreak attacks that disguise harmful intent through indirect or deceptive phrasing. Using causal intervention, we empirically demonstrate that this vulnerability stems from shallow alignment mechanisms that lack deep reasoning, often rejecting harmful prompts without truly understanding why they are harmful. To mitigate this vulnerability, we propose enhancing alignment through reasoning-aware post-training. We construct and release a novel Chain-of-Thought (CoT) fine-tuning dataset that includes both utility-oriented and safety-critical prompts with step-by-step rationales. Fine-tuning on this dataset encourages models to produce principled refusals grounded in reasoning, outperforming standard SFT baselines. Furthermore, inspired by failure patterns in CoT fine-tuning, we introduce Alignment-Weighted DPO, which targets the most problematic parts of an output by assigning different preference weights to the reasoning and final-answer segments. This produces finer-grained, targeted updates than vanilla DPO and improves robustness to diverse jailbreak strategies. Extensive experiments across multiple safety and utility benchmarks show that our method consistently improves alignment robustness while maintaining overall model utility.