I Speak and You Find: Robust 3D Visual Grounding with Noisy and Ambiguous Speech Inputs

TL;DR

SpeechRefer is a novel 3D visual grounding framework that robustly handles noisy and ambiguous speech inputs by integrating acoustic and contrastive modules, significantly improving performance over existing methods.

Contribution

It introduces two innovative modules that reduce reliance on accurate transcriptions, enabling effective 3D visual grounding with imperfect speech-to-text data.

Findings

Significant performance improvements on SpeechRefer and speechNr3D datasets.

Robustness to transcription errors demonstrated in experiments.

Enhanced multimodal system capabilities with noisy speech inputs.

Abstract

Existing 3D visual grounding methods rely on precise text prompts to locate objects within 3D scenes. Speech, as a natural and intuitive modality, offers a promising alternative. Real-world speech inputs, however, often suffer from transcription errors due to accents, background noise, and varying speech rates, limiting the applicability of existing 3DVG methods. To address these challenges, we propose \textbf{SpeechRefer}, a novel 3DVG framework designed to enhance performance in the presence of noisy and ambiguous speech-to-text transcriptions. SpeechRefer integrates seamlessly with xisting 3DVG models and introduces two key innovations. First, the Speech Complementary Module captures acoustic similarities between phonetically related words and highlights subtle distinctions, generating complementary proposal scores from the speech signal. This reduces dependence on potentially erroneous transcriptions. Second, the Contrastive Complementary Module employs contrastive learning to align erroneous text features with corresponding speech features, ensuring robust performance even when transcription errors dominate. Extensive experiments on the SpeechRefer and peechNr3D datasets demonstrate that SpeechRefer improves the performance of existing 3DVG methods by a large margin, which highlights SpeechRefer's potential to bridge the gap between noisy speech inputs and reliable 3DVG, enabling more intuitive and practical multimodal systems.