PREDICT: Preference Reasoning by Evaluating Decomposed preferences Inferred from Candidate Trajectories

TL;DR

PREDICT is a novel method that improves the accuracy of inferring human preferences from trajectories by decomposing preferences, iteratively refining them, and validating across multiple scenarios, outperforming existing methods.

Contribution

The paper introduces PREDICT, a new approach that enhances preference inference accuracy through decomposition, iterative refinement, and validation, addressing limitations of broad and generic preference modeling.

Findings

PREDICT improves preference inference accuracy by 66.2% in gridworld.

PREDICT achieves a 41.0% improvement in the PLUME environment.

The method effectively captures nuanced human preferences.

Abstract

Accommodating human preferences is essential for creating AI agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs to infer preferences from user interactions, but they often produce broad and generic preferences, failing to capture the unique and individualized nature of human preferences. This paper introduces PREDICT, a method designed to enhance the precision and adaptability of inferring preferences. PREDICT incorporates three key elements: (1) iterative refinement of inferred preferences, (2) decomposition of preferences into constituent components, and (3) validation of preferences across multiple trajectories. We evaluate PREDICT on two distinct environments: a gridworld setting and a new text-domain environment (PLUME). PREDICT more accurately infers nuanced human preferences improving over existing baselines by 66.2\% (gridworld environment) and 41.0\% (PLUME).