Unsupervised Human Preference Learning

TL;DR

This paper introduces a novel personalization method for large language models using small preference agent models to generate guiding rules, enabling efficient, fine-tuning-free customization based on individual user preferences.

Contribution

It proposes a new approach where small models act as preference agents to steer larger models, improving personalization without fine-tuning the large model.

Findings

Significantly outperforms baseline personalization methods

Enables data-efficient customization of large language models

Demonstrates effectiveness on email and article datasets

Abstract

Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local "steering wheel" model that directs the outputs of a much larger foundation model, producing content tailored to an individual's preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.