Interactive Language: Talking to Robots in Real Time

TL;DR

This paper introduces a real-time, natural language-interactable robot framework trained on a large dataset, achieving high success in executing diverse commands and guided by human language for complex tasks.

Contribution

It presents a new framework and open-source assets for building interactive robots capable of understanding and executing a wide range of natural language commands in real time.

Findings

93.5% success rate on 87,000 commands

Proficient execution of diverse visuo-linguo-motor skills

Guided by human language for complex, long-horizon tasks

Abstract

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.