Learning Transferable Latent User Preferences for Human-Aligned Decision Making

TL;DR

The paper introduces CLIPR, a framework that learns transferable natural language rules to infer latent user preferences from minimal interactions, enhancing human-aligned decision making in LLMs.

Contribution

It proposes a novel method for inferring and applying latent user preferences using minimal conversational input, improving generalization and reducing inference costs.

Findings

CLIPR outperforms existing methods in alignment accuracy.

CLIPR reduces the number of interactions needed for preference inference.

CLIPR demonstrates effectiveness across multiple datasets and environments.

Abstract

Large language models (LLMs) are increasingly used as reasoning modules in many applications. While they are efficient in certain tasks, LLMs often struggle to produce human-aligned solutions. Human-aligned decision making requires accounting for both explicitly stated goals and latent user preferences that shape how ambiguous situations should be resolved. Existing approaches to incorporating such preferences either rely on extensive and repeated user interactions or fail to generalize latent preferences across tasks and contexts, limiting their practical applicability. We consider a setting in which an LLM is used for high-level reasoning and is responsible for inferring latent user preferences from limited interactions, which guides downstream decision making. We introduce CLIPR (Conversational Learning for Inferring Preferences and Reasoning), a framework that learns actionable, transferable natural language rules that represent latent user preferences from minimal conversational input. These rules are iteratively refined through adaptive feedback and applied to both in-distribution and out-of-distribution ambiguous tasks across multiple environments. Evaluations on three datasets and a user study show that CLIPR consistently outperforms existing methods in improving alignment and reducing inference costs.