CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation

TL;DR

CapNav introduces a benchmark to evaluate vision-language models' ability to perform indoor navigation tasks conditioned on different agent capabilities, revealing current models' limitations in handling mobility constraints and spatial reasoning.

Contribution

This work presents CapNav, a novel benchmark for capability-conditioned indoor navigation, including diverse agents, scenes, and tasks to assess VLMs' spatial reasoning under mobility constraints.

Findings

Current VLMs' navigation performance declines with stricter mobility constraints.

State-of-the-art models struggle with obstacle types requiring spatial reasoning.

CapNav provides a comprehensive platform for evaluating capability-aware embodied navigation.

Abstract

Vision-Language Models (VLMs) have shown remarkable progress in Vision-Language Navigation (VLN), offering new possibilities for navigation decision-making that could benefit both robotic platforms and human users. However, real-world navigation is inherently conditioned by the agent's mobility constraints. For example, a sweeping robot cannot traverse stairs, while a quadruped can. We introduce Capability-Conditioned Navigation (CapNav), a benchmark designed to evaluate how well VLMs can navigate complex indoor spaces given an agent's specific physical and operational capabilities. CapNav defines five representative human and robot agents, each described with physical dimensions, mobility capabilities, and environmental interaction abilities. CapNav provides 45 real-world indoor scenes, 473 navigation tasks, and 2365 QA pairs to test if VLMs can traverse indoor environments based on agent capabilities. We evaluate 13 modern VLMs and find that current VLM's navigation performance drops sharply as mobility constraints tighten, and that even state-of-the-art models struggle with obstacle types that require reasoning on spatial dimensions. We conclude by discussing the implications for capability-aware navigation and the opportunities for advancing embodied spatial reasoning in future VLMs. The benchmark is available at https://github.com/makeabilitylab/CapNav