Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

TL;DR

This paper proposes a selective token alignment method for preference optimization in large language models, improving efficiency and effectiveness by focusing on high-impact tokens during post-training alignment.

Contribution

It introduces a token-level selective alignment strategy that leverages token importance and reference model quality to enhance preference alignment in LLMs.

Findings

Selective alignment reduces computational overhead.

Stronger reference models improve token selection accuracy.

Our method outperforms standard DPO and distillation baselines.

Abstract

Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance. This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs, leveraging token-level log-probability differences between the current policy and a reference model. By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity. We further explore the role of reference model quality, demonstrating that stronger reference models significantly improve token selection accuracy and overall optimization effectiveness. Comprehensive experiments on benchmarks such as Arena-Hard and MT-Bench validate the superiority of our Selective-DPO method over standard DPO and distillation-based baselines. Our findings highlight the importance of token-level optimization and reference model selection in advancing preference alignment for LLMs. The code is available at https://github.com/Dongzhijin/SDPO.