Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression

TL;DR

This paper introduces a steerable pluralistic alignment method for large language models that captures diverse user preferences across multiple attributes using few-shot comparative regression, outperforming existing techniques.

Contribution

The paper proposes a novel few-shot comparative regression approach for pluralistic alignment, enabling LLMs to adapt to individual preferences across multiple attributes with interpretability.

Findings

Outperforms baseline and state-of-the-art methods

Demonstrates applicability to value-aligned decision-making

Provides new benchmarks for pluralistic alignment

Abstract

Large language models (LLMs) are currently aligned using techniques such as reinforcement learning from human feedback (RLHF). However, these methods use scalar rewards that can only reflect user preferences on average. Pluralistic alignment instead seeks to capture diverse user preferences across a set of attributes, moving beyond just helpfulness and harmlessness. Toward this end, we propose a steerable pluralistic model based on few-shot comparative regression that can adapt to individual user preferences. Our approach leverages in-context learning and reasoning, grounded in a set of fine-grained attributes, to compare response options and make aligned choices. To evaluate our algorithm, we also propose two new steerable pluralistic benchmarks by adapting the Moral Integrity Corpus (MIC) and the HelpSteer2 datasets, demonstrating the applicability of our approach to value-aligned decision-making and reward modeling, respectively. Our few-shot comparative regression approach is interpretable and compatible with different attributes and LLMs, while outperforming multiple baseline and state-of-the-art methods. Our work provides new insights and research directions in pluralistic alignment, enabling a more fair and representative use of LLMs and advancing the state-of-the-art in ethical AI.