Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas

TL;DR

This paper introduces a method to infer user personas from preference data in LLMs, enhancing personalization by understanding why users prefer certain responses, and demonstrates improved tailoring of outputs to individual needs.

Contribution

It proposes abductive reasoning to infer user personas from preferences and trains models to personalize responses based on these inferred personas, advancing personalization techniques.

Findings

LLMs can accurately infer user personas explaining preference differences.

Training with persona-augmented data improves personalization and generalizes to new user personas.

Rejection-based personas are more challenging but improve personalization for uncommon preferences.

Abstract

LLMs are aligned to follow input instructions by learning which of two responses users prefer for a prompt. However, such preference data do not convey why users prefer responses that are chosen or rejected, so LLMs trained on these datasets cannot tailor responses to varied user needs. To surface these parameters of personalization, we apply abductive reasoning to preference data, inferring needs and interests of users, i.e., personas, that may prefer either response. We test this idea in two steps: Persona Inference (PI), abductively inferring personas of users who prefer chosen or rejected outputs, and Persona Tailoring (PT), training models to tailor outputs to personas from PI. We show: 1) LLMs infer personas accurately explaining why different users may prefer both chosen or rejected outputs; 2) Training on preference data augmented with PI personas via PT boosts personalization and generalizes to supporting user-written personas; and 3) Rejected response personas form harder personalization evaluations, showing PT better aids users with uncommon preferences versus typical alignment methods. We argue for an abductive view of preferences for personalization, asking not only which response is better but when, why, and for whom.