TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights

TL;DR

This paper introduces TIS-DPO, a token-level importance sampling method for direct preference optimization in LLMs, improving alignment by weighting tokens based on estimated importance, leading to better performance on multiple tasks.

Contribution

It proposes a novel token-level importance sampling approach for DPO, estimating token importance weights using contrastive LLMs to enhance optimization efficiency and alignment results.

Findings

TIS-DPO outperforms baseline methods on alignment tasks.

Estimated token weights effectively identify key token positions.

The method improves optimization efficiency and model alignment quality.

Abstract

Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results. In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance. However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization. Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses. Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions.