Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding

TL;DR

This paper introduces MCM-VG, a novel framework for zero-shot 3D visual grounding that improves accuracy by establishing multiple consistent 2D-3D mappings and leveraging large language and vision models.

Contribution

MCM-VG explicitly enforces 2D-3D consistency across three dimensions to enhance zero-shot 3D visual grounding performance.

Findings

Achieves 62.0% accuracy at 0.25 IoU on ScanRefer, surpassing previous methods.

Sets new state-of-the-art results on ScanRefer and Nr3D benchmarks.

Effectively reconstructs missing targets and reduces spatial redundancy in 3D grounding.

Abstract

Zero-shot 3D Visual Grounding (3DVG) is a critical capability for open-world embodied AI. However, existing methods are fundamentally bottlenecked by the poor quality of open-vocabulary 3D proposals, suffering from inaccurate categories and imprecise geometries, as well as the spatial redundancy of exhaustive multi-view reasoning. To address these challenges, we propose MCM-VG, a novel framework that achieves robust zero-shot 3DVG by explicitly establishing Multiple Consistent 2D-3D Mappings. Instead of passively relying on noisy 3D segments, MCM-VG enforces 2D-3D consistency across three fundamental dimensions to achieve precise target localization and reliable reasoning. First, a Semantic Alignment module corrects category mismatches via LLM-driven query parsing and coarse-to-fine 2D-3D matching. Second, an Instance Rectification module leverages VLM-guided 2D segmentations to reconstruct missing targets, back-projecting these reliable visual priors to establish accurate 3D geometries. Finally, to eliminate spatial redundancy, a Viewpoint Distillation module clusters 3D camera directions to extract optimal frames. By pairing these optimal RGB frames with Bird's Eye View maps into concise visual prompt sets, we formulate the final target disambiguation as a multiple-choice reasoning task for Vision-Language Models. Extensive evaluations on ScanRefer and Nr3D benchmarks demonstrate that MCM-VG sets a new state-of-the-art for zero-shot 3D visual grounding. Remarkably, it achieves 62.0\% and 53.6\% in Acc@0.25 and Acc@0.5 on ScanRefer, outperforming previous baselines by substantial margins of 6.4\% and 4.0\%.