NaviTrace: Evaluating Embodied Navigation of Vision-Language Models

TL;DR

NaviTrace is a new benchmark for evaluating vision-language models' robotic navigation capabilities across diverse scenarios, highlighting current gaps in spatial grounding and goal localization compared to humans.

Contribution

Introduces NaviTrace, a comprehensive benchmark with a novel semantic-aware trace score for assessing embodied navigation of vision-language models.

Findings

Eight state-of-the-art VLMs show significant performance gaps to humans.

The benchmark reveals issues with spatial grounding and goal localization.

NaviTrace provides a scalable, reproducible platform for future research.

Abstract

Vision-language models demonstrate unprecedented performance and generalization across a wide range of tasks and scenarios. Integrating these foundation models into robotic navigation systems opens pathways toward building general-purpose robots. Yet, evaluating these models' navigation capabilities remains constrained by costly real-world trials, overly simplified simulations, and limited benchmarks. We introduce NaviTrace, a high-quality Visual Question Answering benchmark where a model receives an instruction and embodiment type (human, legged robot, wheeled robot, bicycle) and must output a 2D navigation trace in image space. Across 1000 scenarios and more than 3000 expert traces, we systematically evaluate eight state-of-the-art VLMs using a newly introduced semantic-aware trace score. This metric combines Dynamic Time Warping distance, goal endpoint error, and embodiment-conditioned penalties derived from per-pixel semantics and correlates with human preferences. Our evaluation reveals consistent gap to human performance caused by poor spatial grounding and goal localization. NaviTrace establishes a scalable and reproducible benchmark for real-world robotic navigation. The benchmark and leaderboard can be found at https://leggedrobotics.github.io/navitrace_webpage/.