Motif: Intrinsic Motivation from Artificial Intelligence Feedback

TL;DR

Motif introduces a novel approach where an LLM-derived intrinsic reward guides reinforcement learning in complex environments, leading to superior performance and human-aligned behaviors without environment interaction.

Contribution

The paper presents Motif, a method that uses LLMs to generate intrinsic rewards for agents, enabling effective learning in challenging environments without prior environment interaction.

Findings

Motif outperforms direct score maximization in NetHack.

Combining intrinsic and environment rewards improves performance.

Motif behaviors are human-aligned and easily steerable.

Abstract

Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging. In this paper, we propose Motif, a general method to interface such prior knowledge from a Large Language Model (LLM) with an agent. Motif is based on the idea of grounding LLMs for decision-making without requiring them to interact with the environment: it elicits preferences from an LLM over pairs of captions to construct an intrinsic reward, which is then used to train agents with reinforcement learning. We evaluate Motif's performance and behavior on the challenging, open-ended and procedurally-generated NetHack game. Surprisingly, by only learning to maximize its intrinsic reward, Motif achieves a higher game score than an algorithm directly trained to maximize the score itself. When combining Motif's intrinsic reward with the environment reward, our method significantly outperforms existing approaches and makes progress on tasks where no advancements have ever been made without demonstrations. Finally, we show that Motif mostly generates intuitive human-aligned behaviors which can be steered easily through prompt modifications, while scaling well with the LLM size and the amount of information given in the prompt.