GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulation

TL;DR

This paper introduces GVCCI, a lifelong learning framework that enables visual grounding models to adapt to robotic manipulation tasks without human supervision, significantly improving performance across diverse environments.

Contribution

GVCCI is the first lifelong learning approach for visual grounding in robotic manipulation, generating synthetic instructions to continuously improve model accuracy without human labels.

Findings

VG accuracy improved by up to 56.7%

LGRM performance increased by up to 29.4%

Introduced a large-scale dataset with 252k triplets

Abstract

Language-Guided Robotic Manipulation (LGRM) is a challenging task as it requires a robot to understand human instructions to manipulate everyday objects. Recent approaches in LGRM rely on pre-trained Visual Grounding (VG) models to detect objects without adapting to manipulation environments. This results in a performance drop due to a substantial domain gap between the pre-training and real-world data. A straightforward solution is to collect additional training data, but the cost of human-annotation is extortionate. In this paper, we propose Grounding Vision to Ceaselessly Created Instructions (GVCCI), a lifelong learning framework for LGRM, which continuously learns VG without human supervision. GVCCI iteratively generates synthetic instruction via object detection and trains the VG model with the generated data. We validate our framework in offline and online settings across diverse environments on different VG models. Experimental results show that accumulating synthetic data from GVCCI leads to a steady improvement in VG by up to 56.7% and improves resultant LGRM by up to 29.4%. Furthermore, the qualitative analysis shows that the unadapted VG model often fails to find correct objects due to a strong bias learned from the pre-training data. Finally, we introduce a novel VG dataset for LGRM, consisting of nearly 252k triplets of image-object-instruction from diverse manipulation environments.