MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation

TL;DR

MSGNav introduces a multi-modal 3D scene graph for zero-shot embodied navigation, enhancing open vocabulary generalization and visual evidence preservation, leading to state-of-the-art results in benchmark tests.

Contribution

The paper proposes M3DSG to retain visual cues in scene graphs and develops MSGNav with modules for efficient reasoning, open vocabulary support, and last mile problem resolution.

Findings

Achieves state-of-the-art results on GOAT-Bench and HM3D-ObjNav.

Demonstrates effective open vocabulary generalization.

Improves navigation accuracy with visibility-based viewpoint decision.

Abstract

Embodied navigation is a fundamental capability for robotic agents operating. Real-world deployment requires open vocabulary generalization and low training overhead, motivating zero-shot methods rather than task-specific RL training. However, existing zero-shot methods that build explicit 3D scene graphs often compress rich visual observations into text-only relations, leading to high construction cost, irreversible loss of visual evidence, and constrained vocabularies. To address these limitations, we introduce the Multi-modal 3D Scene Graph (M3DSG), which preserves visual cues by replacing textual relational edges with dynamically assigned images. Built on M3DSG, we propose MSGNav, a zero-shot navigation system that includes a Key Subgraph Selection module for efficient reasoning, an Adaptive Vocabulary Update module for open vocabulary support, and a Closed-Loop Reasoning module for accurate exploration reasoning. Additionally, we further identify the last mile problem in zero-shot navigation determining the feasible target location with a suitable final viewpoint, and propose a Visibility-based Viewpoint Decision module to explicitly resolve it. Comprehensive experimental results demonstrate that MSGNav achieves state-of-the-art performance on the challenging GOAT-Bench and HM3D-ObjNav benchmark. The code will be publicly available at https://github.com/ylwhxht/MSGNav.