ICPL: Few-shot In-context Preference Learning via LLMs

TL;DR

This paper introduces ICPL, a novel method leveraging Large Language Models' in-context learning to efficiently learn preferences in reinforcement learning, significantly outperforming baseline methods in both synthetic and human-in-the-loop settings.

Contribution

ICPL is the first approach to utilize LLMs' native preference-learning capabilities for sample-efficient reinforcement learning with human feedback.

Findings

ICPL outperforms baseline preference methods in synthetic tests.

ICPL achieves higher performance and efficiency in preference learning.

ICPL effectively incorporates human feedback in real-world trials.

Abstract

Preference-based reinforcement learning is an effective way to handle tasks where rewards are hard to specify but can be exceedingly inefficient as preference learning is often tabula rasa. We demonstrate that Large Language Models (LLMs) have native preference-learning capabilities that allow them to achieve sample-efficient preference learning, addressing this challenge. We propose In-Context Preference Learning (ICPL), which uses in-context learning capabilities of LLMs to reduce human query inefficiency. ICPL uses the task description and basic environment code to create sets of reward functions which are iteratively refined by placing human feedback over videos of the resultant policies into the context of an LLM and then requesting better rewards. We first demonstrate ICPL's effectiveness through a synthetic preference study, providing quantitative evidence that it significantly outperforms baseline preference-based methods with much higher performance and orders of magnitude greater efficiency. We observe that these improvements are not solely coming from LLM grounding in the task but that the quality of the rewards improves over time, indicating preference learning capabilities. Additionally, we perform a series of real human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop.