Everyone Deserves A Reward: Learning Customized Human Preferences

TL;DR

This paper introduces a method for learning customized reward models that capture diverse human preferences across different domains, improving personalized alignment of language models.

Contribution

It presents a new domain-specific preference dataset and a three-stage learning scheme for personalized reward models, enhancing data efficiency and preference preservation.

Findings

The three-stage scheme improves customization effectiveness.

General preference enrichment helps preserve broad alignment.

Customized preference imitation enhances personalization.

Abstract

Reward models (RMs) are essential for aligning large language models (LLMs) with human preferences to improve interaction quality. However, the real world is pluralistic, which leads to diversified human preferences with respect to different religions, politics, cultures, etc. Moreover, each individual can have their unique preferences on various topics. Neglecting the diversity of human preferences, current human feedback aligning methods only consider a general reward model, which is below satisfaction for customized or personalized application scenarios. To explore customized preference learning, we collect a domain-specific preference (DSP) dataset, which includes preferred responses for each given query from four practical domains. Besides, from the perspective of data efficiency, we propose a three-stage customized RM learning scheme, then empirically verify its effectiveness on both general preference datasets and our DSP set. Furthermore, we test multiple training and data strategies on the three learning stages. We find several ways to better preserve the general preferring ability while training the customized RMs, especially general preference enrichment, and customized preference imitation learning. The DSP dataset and code are available at https://github.com/Linear95/DSP.