BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization

TL;DR

This paper introduces BAPO, a method for personalizing large language models that preserves general knowledge while adapting to diverse user preferences, addressing the challenge of knowledge forgetting in personalized alignment.

Contribution

BAPO is a novel approach that uses reference model responses to prevent forgetting during personalized preference optimization in LLMs.

Findings

BAPO effectively balances personalization and knowledge preservation.

It outperforms previous methods in diverse preference adaptation.

BAPO maintains global knowledge with minimal loss.

Abstract

While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain general knowledge and alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment. Our experiments demonstrate the efficacy of BAPO in various setups.