Debiasing Online Preference Learning via Preference Feature Preservation

TL;DR

This paper introduces PFP, a framework that preserves human preference features during online learning to reduce bias and improve alignment of large language models.

Contribution

The paper proposes a novel Preference Feature Preservation framework that maintains human preference feature distributions during online learning, enhancing bias mitigation and model alignment.

Findings

PFP effectively reduces bias in preference features during online learning.

PFP outperforms previous methods on standard LLM alignment benchmarks.

The framework improves the handling of human preferences in LLM responses.

Abstract

Recent preference learning frameworks for large language models (LLMs) simplify human preferences with binary pairwise comparisons and scalar rewards. This simplification could make LLMs' responses biased to mostly preferred features, and would be exacerbated during the iterations of online preference learning steps. To address these challenges, we propose a novel framework coined PFP (Preference Feature Preservation). The key idea of PFP is maintaining the distribution of human preference features and utilizing such rich signals throughout the online preference learning process. Specifically, PFP first extract preference features from offline pairwise human preference data and trains a feature classifier. Then, using trained classifier and the distribution preserving optimization, PFP maps appropriate preference features for a new input instruction during online learning. Lastly, PFP trains LLM using the existing preference learning method, by incorporating the preference feature into system prompts and enabling LLM to explicitly handle various human preferences. Our experiments demonstrate that PFP successfully mitigates the bias in preference features during online learning, and hence achieves superior performance compared to previous preference learning methods on standard benchmarks to evaluate LLM alignment.