LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models

TL;DR

This paper introduces LangGap, a benchmark designed to diagnose and improve language understanding in vision-language-action models by using semantic perturbations and diverse tasks, revealing significant gaps in current models.

Contribution

The paper presents LangGap, a novel benchmark with semantic perturbations and diverse tasks to systematically evaluate and diagnose language understanding deficits in VLA models.

Findings

Targeted data augmentation improves success rate from 0% to 90%.

Multi-task training increases success rate from 0% to 28%.

Models struggle with increased semantic diversity, revealing fundamental limitations.

Abstract

Vision-Language-Action (VLA) models achieve over 95% success on standard benchmarks. However, through systematic experiments, we find that current state-of-the-art VLA models largely ignore language instructions. Prior work lacks: (1) systematic semantic perturbation diagnostics, (2) a benchmark that forces language understanding by design, and (3) linguistically diverse training data. This paper constructs the LangGap benchmark, based on a four-dimensional semantic perturbation method -- varying instruction semantics while keeping the tabletop layout fixed -- revealing language understanding deficits in {\pi}0.5. Existing benchmarks like LIBERO assign only one task per layout, underutilizing available objects and target locations; LangGap fully diversifies pick-and-place tasks under identical layouts, forcing models to truly understand language. Experiments show that targeted data augmentation can partially close the language gap -- success rate improves from 0% to 90% with single-task training, and 0% to 28% with multi-task training. However, as semantic diversity of extended tasks increases, model learning capacity proves severely insufficient; even trained tasks perform poorly. This reveals a fundamental challenge for VLA models in understanding diverse language instructions -- precisely the long-term value of LangGap.