Averaging log-likelihoods in direct alignment

TL;DR

This paper proposes a length-invariant averaging method for direct alignment of Large Language Models, improving alignment with human preferences by addressing length bias in log-likelihood comparisons.

Contribution

It introduces a novel averaging operator for length-invariance in direct alignment, bridging contrastive and supervised training approaches.

Findings

Averaging log-likelihoods affects generation scores based on length.

The method reveals a trade-off between generation length and quality.

Empirical results demonstrate improved alignment consistency.

Abstract

To better align Large Language Models (LLMs) with human judgment, Reinforcement Learning from Human Feedback (RLHF) learns a reward model and then optimizes it using regularized RL. Recently, direct alignment methods were introduced to learn such a fine-tuned model directly from a preference dataset without computing a proxy reward function. These methods are built upon contrastive losses involving the log-likelihood of (dis)preferred completions according to the trained model. However, completions have various lengths, and the log-likelihood is not length-invariant. On the other side, the cross-entropy loss used in supervised training is length-invariant, as batches are typically averaged token-wise. To reconcile these approaches, we introduce a principled approach for making direct alignment length-invariant. Formally, we introduce a new averaging operator, to be composed with the optimality operator giving the best policy for the underlying RL problem. It translates into averaging the log-likelihood within the loss. We empirically study the effect of such averaging, observing a trade-off between the length of generations and their scores.