Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

TL;DR

This paper introduces SPA, a novel framework that enhances large language model alignment using minimal human preference data by leveraging model logits and self-annotated learning, significantly reducing annotation costs.

Contribution

SPA is a new framework that improves LLM alignment efficiently by deriving preferences from model logits and iteratively self-improving with minimal human annotations.

Findings

Achieves superior alignment with only 3.3% of ground-truth labels.

Outperforms state-of-the-art baselines on AlpacaEval 2.0.

Effectively mitigates low-quality data with noise-aware learning.

Abstract

Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.