PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimization

TL;DR

PICACO is a novel method for pluralistic in-context alignment of LLMs that optimizes total correlation to better understand and balance multiple human values without fine-tuning.

Contribution

It introduces a new ICA approach that maximizes total correlation between multiple values and LLM responses, addressing the instruction bottleneck challenge.

Findings

Outperforms recent baselines on five value sets

Works well with both black-box and open-source LLMs

Achieves better balance across up to 8 values

Abstract

In-Context Learning has shown great potential for aligning Large Language Models (LLMs) with human values, helping reduce harmful outputs and accommodate diverse preferences without costly post-training, known as In-Context Alignment (ICA). However, LLMs' comprehension of input prompts remains agnostic, limiting ICA's ability to address value tensions--human values are inherently pluralistic, often imposing conflicting demands, e.g., stimulation vs. tradition. Current ICA methods therefore face the Instruction Bottleneck challenge, where LLMs struggle to reconcile multiple intended values within a single prompt, leading to incomplete or biased alignment. To address this, we propose PICACO, a novel pluralistic ICA method. Without fine-tuning, PICACO optimizes a meta-instruction that navigates multiple values to better elicit LLMs' understanding of them and improve their alignment. This is achieved by maximizing the total correlation between specified values and LLM responses, theoretically reinforcing value correlation while reducing distractive noise, resulting in effective value instructions. Extensive experiments on five value sets show that PICACO works well with both black-box and open-source LLMs, outperforms several recent strong baselines, and achieves a better balance across up to 8 distinct values.