Beyond Waypoints: Dual-Heatmap Grounding for Cross-Embodiment Semantic Navigation

TL;DR

This paper introduces a dual-heatmap approach for semantic navigation that predicts continuous reachable regions and orientation constraints, improving robustness and transferability across different robot embodiments.

Contribution

It proposes a unified vision-language framework using dual heatmaps to better model spatial uncertainty and improve navigation success in human-robot interaction.

Findings

Achieves state-of-the-art performance among 8B parameter models.

Explicit heatmap prediction significantly improves affordance rate.

Demonstrates robustness across diverse robot embodiments.

Abstract

Grounding open-ended semantic instructions into physically executable local goals is a fundamental challenge in human-robot interaction. While existing navigation frameworks often regress deterministic waypoints, this rigid formulation collapses spatial uncertainty and frequently targets non-traversable object centers, leading to severe execution failures. In this work, we focus on the practical setting of in-FOV semantic navigation, where a robot receives concise, interleaved multimodal (text and image) prompts. To bridge the gap between abstract semantic intent and physical reachability, we propose a unified Vision-Language framework that abandons single-point regression in favor of a Dual-Heatmap representation. Our framework predicts a navigation affordance heatmap that captures continuous reachable regions, coupled with a facing heatmap for orientation constraints. These dense outputs inherently function as a differentiable semantic potential field, integrating seamlessly with downstream local planners. To support this paradigm, we build a fully automated, foundation-model-assisted synthetic data pipeline and establish a comprehensive simulation benchmark. Extensive experiments demonstrate that our framework achieves state-of-the-art performance among comparable 8B baselines. Crucially, a feature-fusion study and simulation studies across diverse robot embodiments (Jetbot, H1, Aliengo) reveal that explicit heatmap prediction drastically improves the Affordance Rate (AR). By placing targets reliably in executable free space, our framework effectively mitigates the brittleness of point regression, offering a transferable path toward safe cross-embodiment semantic navigation.