Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning

TL;DR

Robo-MUTUAL introduces a novel approach for robotic multimodal task specification that leverages unimodal data and out-of-domain pretraining to achieve effective cross-modality alignment, enabling robots to understand complex instructions with limited paired data.

Contribution

The paper presents a method to teach robots multimodal task understanding using unimodal data and novel alignment techniques, reducing the need for extensive paired multimodal datasets.

Findings

Outperforms existing methods on the LIBERO benchmark.

Successfully generalizes to over 130 tasks and real robot platforms.

Achieves significant improvements in data efficiency for robotic learning.

Abstract

Multimodal task specification is essential for enhanced robotic performance, where \textit{Cross-modality Alignment} enables the robot to holistically understand complex task instructions. Directly annotating multimodal instructions for model training proves impractical, due to the sparsity of paired multimodal data. In this study, we demonstrate that by leveraging unimodal instructions abundant in real data, we can effectively teach robots to learn multimodal task specifications. First, we endow the robot with strong \textit{Cross-modality Alignment} capabilities, by pretraining a robotic multimodal encoder using extensive out-of-domain data. Then, we employ two Collapse and Corrupt operations to further bridge the remaining modality gap in the learned multimodal representation. This approach projects different modalities of identical task goal as interchangeable representations, thus enabling accurate robotic operations within a well-aligned multimodal latent space. Evaluation across more than 130 tasks and 4000 evaluations on both simulated LIBERO benchmark and real robot platforms showcases the superior capabilities of our proposed framework, demonstrating significant advantage in overcoming data constraints in robotic learning. Website: zh1hao.wang/Robo_MUTUAL