Inferring Rewards from Language in Context

TL;DR

This paper introduces a model that infers user reward functions from language pragmatically, enabling better action prediction in new contexts by understanding underlying preferences expressed through language.

Contribution

The paper presents a novel approach that directly infers rewards from language pragmatically, improving over traditional methods that separate language-to-action and action-to-reward mappings.

Findings

More accurate reward inference from language in new environments

Improved prediction of optimal actions in unseen contexts

Outperforms previous instruction-following and inverse reinforcement learning methods

Abstract

In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).