Efficient Personalization of Generative Models via Optimal Experimental Design

TL;DR

This paper introduces an optimal experimental design approach to efficiently personalize generative models by selecting the most informative human preference queries, reducing the amount of feedback needed for effective customization.

Contribution

It formulates preference query selection as a convex optimization problem and proposes ED-PBRL, an efficient algorithm with theoretical guarantees for structured query construction.

Findings

Requires fewer preference queries than random selection

Effectively personalizes text-to-image models to user styles

Provides a convex optimization framework with theoretical support

Abstract

Preference learning from human feedback has the ability to align generative models with the needs of end-users. Human feedback is costly and time-consuming to obtain, which creates demand for data-efficient query selection methods. This work presents a novel approach that leverages optimal experimental design to ask humans the most informative preference queries, from which we can elucidate the latent reward function modeling user preferences efficiently. We formulate the problem of preference query selection as the one that maximizes the information about the underlying latent preference model. We show that this problem has a convex optimization formulation, and introduce a statistically and computationally efficient algorithm ED-PBRL that is supported by theoretical guarantees and can efficiently construct structured queries such as images or text. We empirically present the proposed framework by personalizing a text-to-image generative model to user-specific styles, showing that it requires less preference queries compared to random query selection.