MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

TL;DR

MAPLE is a novel framework that uses large language models to guide Bayesian active preference learning, reducing human effort and improving learning efficiency through natural language feedback and active query selection.

Contribution

It introduces a new LLM-guided Bayesian active preference learning framework that models preference functions and optimizes query selection for efficiency and interpretability.

Findings

MAPLE accelerates preference learning compared to baseline methods.

It reduces human supervision effort in preference elicitation.

MAPLE achieves high-quality preference inference on real-world benchmarks.

Abstract

The advent of large language models (LLMs) has sparked significant interest in using natural language for preference learning. However, existing methods often suffer from high computational burdens, taxing human supervision, and lack of interpretability. To address these issues, we introduce MAPLE, a framework for large language model-guided Bayesian active preference learning. MAPLE leverages LLMs to model the distribution over preference functions, conditioning it on both natural language feedback and conventional preference learning feedback, such as pairwise trajectory rankings. MAPLE also employs active learning to systematically reduce uncertainty in this distribution and incorporates a language-conditioned active query selection mechanism to identify informative and easy-to-answer queries, thus reducing human burden. We evaluate MAPLE's sample efficiency and preference inference quality across two benchmarks, including a real-world vehicle route planning benchmark using OpenStreetMap data. Our results demonstrate that MAPLE accelerates the learning process and effectively improves humans' ability to answer queries.