Language-guided Task Adaptation for Imitation Learning

TL;DR

This paper presents a new setting where agents learn tasks from demonstrations of related tasks using natural language, enabling flexible task adaptation and correction, supported by new benchmarks and a transformer-based reasoning framework.

Contribution

It introduces a novel language-guided task adaptation setting, along with two benchmarks and a transformer-based model for reasoning about task entities and relationships.

Findings

The framework effectively adapts to new tasks using natural language descriptions.

Benchmarks demonstrate the approach's versatility across diverse task types.

Results show improved task learning efficiency with language guidance.

Abstract

We introduce a novel setting, wherein an agent needs to learn a task from a demonstration of a related task with the difference between the tasks communicated in natural language. The proposed setting allows reusing demonstrations from other tasks, by providing low effort language descriptions, and can also be used to provide feedback to correct agent errors, which are both important desiderata for building intelligent agents that assist humans in daily tasks. To enable progress in this proposed setting, we create two benchmarks -- Room Rearrangement and Room Navigation -- that cover a diverse set of task adaptations. Further, we propose a framework that uses a transformer-based model to reason about the entities in the tasks and their relationships, to learn a policy for the target task