Learning to Follow Language Instructions with Compositional Policies

TL;DR

This paper introduces a framework that combines compositional value functions and language understanding to enable reinforcement learning agents to efficiently learn and generalize to new goal-reaching tasks with fewer training steps.

Contribution

It presents a novel approach that leverages compositionality in both value functions and language to reduce sample complexity in learning new tasks.

Findings

86% reduction in training steps for new tasks

Effective generalization to unseen tasks through compositional policies

Combines reinforcement learning with language-to-logical expression mapping

Abstract

We propose a framework that learns to execute natural language instructions in an environment consisting of goal-reaching tasks that share components of their task descriptions. Our approach leverages the compositionality of both value functions and language, with the aim of reducing the sample complexity of learning novel tasks. First, we train a reinforcement learning agent to learn value functions that can be subsequently composed through a Boolean algebra to solve novel tasks. Second, we fine-tune a seq2seq model pretrained on web-scale corpora to map language to logical expressions that specify the required value function compositions. Evaluating our agent in the BabyAI domain, we observe a decrease of 86% in the number of training steps needed to learn a second task after mastering a single task. Results from ablation studies further indicate that it is the combination of compositional value functions and language representations that allows the agent to quickly generalize to new tasks.