Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift

TL;DR

This paper introduces NS-DPO, a novel method for fine-tuning large language models that accounts for changing user preferences over time, improving robustness and alignment in dynamic environments.

Contribution

The paper proposes a computationally efficient non-stationary preference optimization method using a dynamic model and theoretical analysis of its convergence and error bounds.

Findings

NS-DPO outperforms baseline algorithms under preference drift.

NS-DPO maintains robustness in non-stationary preference scenarios.

The method does not sacrifice performance in stationary cases.

Abstract

Current Large Language Model (LLM) preference optimization algorithms do not account for temporal preference drift, which can lead to severe misalignment. To address this limitation, we propose Non-Stationary Direct Preference Optimisation (NS-DPO) that models time-dependent reward functions with a Dynamic Bradley-Terry model. NS-DPO proposes a computationally efficient solution by introducing only a single discount parameter in the loss function, which is used for exponential weighting that proportionally focuses learning on more time-relevant datapoints. We theoretically analyze the convergence of NS-DPO in a general setting where the exact nature of the preference drift is not known, providing upper bounds on the estimation error and regret caused by non-stationary preferences. Finally, we demonstrate the effectiveness of NS-DPO for fine-tuning LLMs under drifting preferences. Using scenarios where various levels of preference drift is introduced, with popular LLM reward models and datasets, we show that NS-DPO fine-tuned LLMs remain robust under non-stationarity, significantly outperforming baseline algorithms that ignore temporal preference changes, without sacrificing performance in stationary cases.