GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents

TL;DR

GROOT-2 introduces a semi-supervised, multimodal instruction-following agent that leverages weak supervision and latent variable models to learn from unlabeled demonstrations and align with human intentions, improving performance across diverse environments.

Contribution

The paper presents GROOT-2, a novel semi-supervised approach combining weak supervision with latent models for multimodal instruction following in robotics and AI.

Findings

Effective in four diverse environments

Learns from unlabeled demonstrations

Aligns latent space with human intentions

Abstract

Developing agents that can follow multimodal instructions remains a fundamental challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets (no language instruction) has enabled agents to learn diverse behaviors, these agents often struggle with following instructions. While augmenting the dataset with instruction labels can mitigate this issue, acquiring such high-quality annotations at scale is impractical. To address this issue, we frame the problem as a semi-supervised learning task and introduce GROOT-2, a multimodal instructable agent trained using a novel approach that combines weak supervision with latent variable models. Our method consists of two key components: constrained self-imitating, which utilizes large amounts of unlabeled demonstrations to enable the policy to learn diverse behaviors, and human intention alignment, which uses a smaller set of labeled demonstrations to ensure the latent space reflects human intentions. GROOT-2's effectiveness is validated across four diverse environments, ranging from video games to robotic manipulation, demonstrating its robust multimodal instruction-following capabilities.