Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment

TL;DR

This paper introduces a novel training framework for hierarchical vision-language-action models that explicitly aligns language descriptions with visual observations and actions, improving robot transparency and human-robot collaboration.

Contribution

It proposes a contrastive learning approach for explicit language-action alignment in hierarchical VLA models, reducing reliance on extensive annotations.

Findings

Achieves performance comparable to fully supervised fine-tuning.

Provides insights into multimodal grounding representations.

Establishes a strong baseline with minimal data annotations.

Abstract

Achieving robot transparency is a critical step toward effective human-robot collaboration. To be transparent, a robot's natural language communication must be consistent with its actions and explicitly grounded in the task and environment. Existing hierarchical Vision-Language-Action (VLA) models can generate language (e.g., through chain-of-thought) and low-level actions. However, current work does not consider explicit alignment between these modalities during training. To address this crucial gap, we propose a novel training framework that explicitly grounds hierarchical VLA sub-task descriptions with respect to the visual observation and action space. Our framework uses a contrastive model to assess the alignment between generated language and corresponding action trajectories. This contrastive model enables direct ranking of different language-trajectory pairs based on their alignment, allowing us to refine the grounding of our hierarchical VLA through offline preference learning. We apply our framework to the LanguageTable dataset, a benchmark dataset of human language-annotated trajectories, and provide critical insights into multimodal grounding representations, all while establishing a strong baseline that achieves performance comparable to fully supervised fine-tuning and minimizing the need for costly data annotations.