Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

TL;DR

This paper introduces a temporal decay mechanism into Direct Preference Optimization (DPO) to prioritize earlier tokens in sequences, improving alignment and performance of large language models across various benchmarks.

Contribution

It proposes a novel temporal decay factor in DPO that dynamically weights rewards based on token position, addressing length bias and enhancing model alignment.

Findings

Outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2

Improves scores by 3.3-9.7 points on Arena-Hard

Enhances performance on mathematical and reasoning benchmarks

Abstract

Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the reference model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking temporal dynamics. To this end, we propose an enhanced preference optimization method that incorporates a temporal decay factor controlled by a gamma parameter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our approach mitigates overfitting to less pertinent data and remains responsive to evolving human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. Furthermore, additional experiments on mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available at \url{https://github.com/LotuSrc/D2PO}.