Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment

TL;DR

This paper proposes a new method for efficiently aligning large language models with individual human preferences by disentangling preference representation from text generation, significantly reducing training time.

Contribution

The paper introduces a flexible paradigm that improves alignment efficiency by separating preference encoding from text generation in LLMs, outperforming existing PEFT-based methods.

Findings

Achieves comparable or better alignment quality than PEFT-based methods.

Reduces additional training time for new preferences by 80-90%.

Validated across multiple text generation tasks.

Abstract

Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by $80\%$ to $90\%$ in comparison with them.