DDAVS: Disentangled Audio Semantics and Delayed Bidirectional Alignment for Audio-Visual Segmentation

TL;DR

DDAVS introduces a novel framework for audio-visual segmentation that disentangles audio semantics and improves alignment, effectively addressing multi-source entanglement and misalignment issues to enhance segmentation accuracy.

Contribution

The paper proposes DDAVS, a new method that uses learnable queries and delayed bidirectional attention to disentangle audio semantics and improve multimodal alignment in AVS.

Findings

Outperforms existing methods on AVS-Objects and VPO benchmarks.

Demonstrates robustness in single-source, multi-source, and multi-instance scenarios.

Achieves superior segmentation accuracy and generalization.

Abstract

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by jointly leveraging auditory and visual information. However, existing methods often suffer from multi-source entanglement and audio-visual misalignment, which lead to biases toward louder or larger objects while overlooking weaker, smaller, or co-occurring sources. To address these challenges, we propose DDAVS, a Disentangled Audio Semantics and Delayed Bidirectional Alignment framework. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross-attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS consistently outperforms existing approaches, exhibiting strong performance across single-source, multi-source, and multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/