Learning Visual Affordance from Audio

TL;DR

This paper introduces a new task called Audio-Visual Affordance Grounding (AV-AG) that segments interaction regions from sounds, along with a dataset and a model that outperform existing methods in understanding object affordances using audio cues.

Contribution

The paper presents the first AV-AG dataset with pixel-level annotations and an unseen subset for zero-shot evaluation, and proposes AVAGFormer, a novel model that effectively fuses audio and visual information for affordance grounding.

Findings

AVAGFormer achieves state-of-the-art results on AV-AG.

The dataset includes a zero-shot subset for generalization testing.

End-to-end modeling improves affordance segmentation performance.

Abstract

We introduce Audio-Visual Affordance Grounding (AV-AG), a new task that segments object interaction regions from action sounds. Unlike existing approaches that rely on textual instructions or demonstration videos, which often limited by ambiguity or occlusion, audio provides real-time, semantically rich, and visually independent cues for affordance grounding, enabling more intuitive understanding of interaction regions. To support this task, we construct the first AV-AG dataset, comprising a large collection of action sounds, object images, and pixel-level affordance annotations. The dataset also includes an unseen subset to evaluate zero-shot generalization. Furthermore, we propose AVAGFormer, a model equipped with a semantic-conditioned cross-modal mixer and a dual-head decoder that effectively fuses audio and visual signals for mask prediction. Experiments show that AVAGFormer achieves state-of-the-art performance on AV-AG, surpassing baselines from related tasks. Comprehensive analyses highlight the distinctions between AV-AG and AVS, the benefits of end-to-end modeling, and the contribution of each component. Code and dataset have been released on https://jscslld.github.io/AVAGFormer/.