Aligning Language Models Using Follow-up Likelihood as Reward Signal

TL;DR

This paper introduces a novel reward signal based on follow-up utterance likelihood to improve language model alignment without human annotations, achieving competitive performance on preference benchmarks.

Contribution

It proposes Follow-up Likelihood as Reward (FLR), a new method for reward modeling that leverages follow-up utterance likelihood, and demonstrates its effectiveness in aligning language models.

Findings

FLR matches strong reward models on preference benchmarks.

Mining preference data from model generations boosts helpfulness.

Fine-tuning models with natural language feedback enhances FLR performance.

Abstract

In natural human-to-human conversations, participants often receive feedback signals from one another based on their follow-up reactions. These reactions can include verbal responses, facial expressions, changes in emotional state, and other non-verbal cues. Similarly, in human-machine interactions, the machine can leverage the user's follow-up utterances as feedback signals to assess whether it has appropriately addressed the user's request. Therefore, we propose using the likelihood of follow-up utterances as rewards to differentiate preferred responses from less favored ones, without relying on human or commercial LLM-based preference annotations. Our proposed reward mechanism, ``Follow-up Likelihood as Reward" (FLR), matches the performance of strong reward models trained on large-scale human or GPT-4 annotated data on 8 pairwise-preference and 4 rating-based benchmarks. Building upon the FLR mechanism, we propose to automatically mine preference data from the online generations of a base policy model. The preference data are subsequently used to boost the helpfulness of the base model through direct alignment from preference (DAP) methods, such as direct preference optimization (DPO). Lastly, we demonstrate that fine-tuning the language model that provides follow-up likelihood with natural language feedback significantly enhances FLR's performance on reward modeling benchmarks and effectiveness in aligning the base policy model's helpfulness.