Reward Design with Language Models

TL;DR

This paper proposes a novel method for reward design in reinforcement learning by using large language models as proxy reward functions, enabling natural language-based reward specification and training of aligned agents.

Contribution

It introduces a framework that leverages LLMs for reward design, simplifying the process with natural language prompts and demonstrating effectiveness across multiple strategic tasks.

Findings

RL agents trained with LLM-based rewards align well with user objectives

The approach outperforms supervised reward learning in various tasks

Natural language prompts effectively guide agent behavior

Abstract

Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning