Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation

TL;DR

Gondola is a grounded vision-language planning model that leverages multi-view images and LLMs to enhance robotic manipulation generalization across unseen objects, environments, and complex tasks.

Contribution

It introduces a novel multi-view grounded planning approach with new datasets, improving LLM-based robotic manipulation in diverse scenarios.

Findings

Gondola outperforms previous LLM-based methods on GemBench across all generalization levels.

It effectively uses multi-view images and segmentation masks for precise object grounding.

The model demonstrates strong generalization to novel objects, placements, and complex tasks.

Abstract

Robotic manipulation faces a significant challenge in generalizing across unseen objects, environments and tasks specified by diverse language instructions. To improve generalization capabilities, recent research has incorporated large language models (LLMs) for planning and action execution. While promising, these methods often fall short in generating grounded plans in visual environments. Although efforts have been made to perform visual instructional tuning on LLMs for robotic manipulation, existing methods are typically constrained by single-view image input and struggle with precise object grounding. In this work, we introduce Gondola, a novel grounded vision-language planning model based on LLMs for generalizable robotic manipulation. Gondola takes multi-view images and history plans to produce the next action plan with interleaved texts and segmentation masks of target objects and locations. To support the training of Gondola, we construct three types of datasets using the RLBench simulator, namely robot grounded planning, multi-view referring expression and pseudo long-horizon task datasets. Gondola outperforms the state-of-the-art LLM-based method across all four generalization levels of the GemBench dataset, including novel placements, rigid objects, articulated objects and long-horizon tasks.