Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

TL;DR

This paper introduces Latent Preference Coding (LPC), a novel framework that models complex human preferences in large language models using discrete latent codes, improving alignment robustness without pre-defined reward functions.

Contribution

LPC provides a unified, data-driven approach to model multifaceted human preferences, enhancing alignment methods without relying on explicit reward functions or hand-crafted weights.

Findings

LPC improves alignment performance across multiple benchmarks.

Latent codes capture differences in human preference distributions.

LPC enhances robustness of alignment against data noise.

Abstract

Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-8B-Instruct). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for the responsible deployment of powerful LLMs.