FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data

TL;DR

This paper introduces FaST, a parameter-efficient method for personalizing large language models to individual user preferences using limited data, supported by new datasets and comprehensive benchmarking.

Contribution

The paper presents FaST, a novel feature-aware tuning approach that effectively personalizes LLMs with minimal data, and provides new datasets for benchmarking.

Findings

FaST outperforms existing methods in personalization tasks.

New datasets DnD and ELIP facilitate research in limited-data personalization.

FaST achieves high performance with fewer parameters.

Abstract

LLM-powered conversational assistants are often deployed in a one-size-fits-all manner, which fails to accommodate individual user preferences. Recently, LLM personalization -- tailoring models to align with specific user preferences -- has gained increasing attention as a way to bridge this gap. In this work, we specifically focus on a practical yet challenging setting where only a small set of preference annotations can be collected per user -- a problem we define as Personalized Preference Alignment with Limited Data (PPALLI). To support research in this area, we introduce two datasets -- DnD and ELIP -- and benchmark a variety of alignment techniques on them. We further propose FaST, a highly parameter-efficient approach that leverages high-level features automatically discovered from the data, achieving the best overall performance.