TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models

TL;DR

TAViS introduces a novel framework that effectively aligns audio and visual data for segmentation by leveraging foundation models and a text-bridged design, improving performance across various datasets and zero-shot scenarios.

Contribution

The paper proposes a new cross-modal alignment method using foundation models and a text-bridged prompting mechanism for audio-visual segmentation.

Findings

Superior performance on multiple datasets

Effective zero-shot segmentation results

Enhanced cross-modal alignment accuracy

Abstract

Audio-Visual Segmentation (AVS) faces a fundamental challenge of effectively aligning audio and visual modalities. While recent approaches leverage foundation models to address data scarcity, they often rely on single-modality knowledge or combine foundation models in an off-the-shelf manner, failing to address the cross-modal alignment challenge. In this paper, we present TAViS, a novel framework that \textbf{couples} the knowledge of multimodal foundation models (ImageBind) for cross-modal alignment and a segmentation foundation model (SAM2) for precise segmentation. However, effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision. To address these challenges, we introduce a text-bridged design with two key components: (1) a text-bridged hybrid prompting mechanism where pseudo text provides class prototype information while retaining modality-specific details from both audio and visual inputs, and (2) an alignment supervision strategy that leverages text as a bridge to align shared semantic concepts within audio-visual modalities. Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.