Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

TL;DR

Instruct2Act leverages large language models to translate multi-modal instructions into robotic actions by generating Python programs, integrating foundation models for perception, and demonstrating superior zero-shot performance in tabletop manipulation tasks.

Contribution

This work introduces a flexible framework that maps multi-modal instructions to robotic actions using LLMs and foundation models, advancing zero-shot capabilities in manipulation tasks.

Findings

Outperforms state-of-the-art policies in several tasks

Effective in zero-shot scenarios

Flexible multi-modal instruction handling

Abstract

Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing. This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks. Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks. In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them. In this way, the framework leverages the expertise of foundation models and robotic abilities to convert complex high-level instructions into precise policy codes. Our approach is adjustable and flexible in accommodating various instruction modalities and input types and catering to specific task demands. We validated the practicality and efficiency of our approach by assessing it on robotic tasks in different scenarios within tabletop manipulation domains. Furthermore, our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks. The code for our proposed approach is available at https://github.com/OpenGVLab/Instruct2Act, serving as a robust benchmark for high-level robotic instruction tasks with assorted modality inputs.