FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users

TL;DR

FSPO is a few-shot learning algorithm that personalizes large language models by quickly inferring user-specific reward functions from minimal preferences, using synthetic data for training.

Contribution

The paper introduces FSPO, a novel meta-learning approach for LLM personalization that leverages synthetic preference data and a new user description rationalization technique.

Findings

FSPO achieves 87% winrate in synthetic user personalization.

FSPO attains 70% winrate with real users in open-ended QA.

Synthetic data with high diversity and coherence is crucial for transfer.

Abstract

Effective personalization of LLMs is critical for a broad range of user-interfacing applications such as virtual assistants and content curation. Inspired by the strong in-context capabilities of LLMs, we propose few-shot preference optimization (FSPO), an algorithm for LLM personalization that reframes reward modeling as a meta-learning problem. Under FSPO, an LLM learns to quickly infer a personalized reward function for a user via a few labeled preferences. FSPO also utilizes user description rationalization (RAT) to encourage better reward modeling and instruction following, recovering performance with the oracle user description. Since real-world preference data is challenging to collect at scale, we propose careful design choices to construct synthetic preference datasets for personalization, generating over 1M synthetic personalized preferences using publicly available LLMs. To successfully transfer from synthetic data to real users, we find it crucial for the data to exhibit both high diversity and coherent, self-consistent structure. We evaluate FSPO on personalized open-ended generation for up to 1,500 synthetic users across three domains: movie reviews, education, and open-ended question answering. We also run a controlled human study. Overall, FSPO achieves an 87% Alpaca Eval winrate in generating responses that are personalized to synthetic users and a 70% winrate with real human users in open-ended question answering.