Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation

TL;DR

This paper investigates direct audio-visual alignment for object grounding, proposing a new dataset and benchmarking models, showing that direct audio grounding can be more robust and efficient than transcription-based methods.

Contribution

It introduces a novel audio-based grounding dataset and benchmarks models, demonstrating the feasibility and advantages of direct audio-visual grounding over traditional transcription pipelines.

Findings

Direct audio grounding is feasible and sometimes superior to transcription-based methods.

Models trained on audio can be more robust to linguistic variability.

The new dataset covers diverse objects and accents, supporting broader research.

Abstract

Understanding human instructions is essential for enabling smooth human-robot interaction. In this work, we focus on object grounding, i.e., localizing an object of interest in a visual scene (e.g., an image) based on verbal human instructions. Despite recent progress, a dominant research trend relies on using text as an intermediate representation. These approaches typically transcribe speech to text, extract relevant object keywords, and perform grounding using models pretrained on large text-vision datasets. However, we question both the efficiency and robustness of such transcription-based pipelines. Specifically, we ask: Can we achieve direct audio-visual alignment without relying on text? To explore this possibility, we simplify the task by focusing on grounding from single-word spoken instructions. We introduce a new audio-based grounding dataset that covers a wide variety of objects and diverse human accents. We then adapt and benchmark several models from the closely audio-visual field. Our results demonstrate that direct grounding from audio is not only feasible but, in some cases, even outperforms transcription-based methods, especially in terms of robustness to linguistic variability. Our findings encourage a renewed interest in direct audio grounding and pave the way for more robust and efficient multimodal understanding systems.