Audio Visual Segmentation Through Text Embeddings

TL;DR

This paper introduces AV2T-SAM, a novel framework that enhances audio-visual segmentation by aligning audio features with text embeddings from pre-trained models, leveraging multimodal data to improve segmentation accuracy.

Contribution

The paper proposes AV2T-SAM, which bridges audio features with text embeddings in pre-trained segmentation models, addressing data scarcity and improving AVS performance.

Findings

Outperforms existing methods on AVSBench dataset

Effectively utilizes pre-trained segmentation models and cross-modal semantic alignment

Introduces a new feature emphasizing shared semantics of audio and visual modalities

Abstract

The goal of Audio-Visual Segmentation (AVS) is to localize and segment the sounding source objects from video frames. Research on AVS suffers from data scarcity due to the high cost of fine-grained manual annotations. Recent works attempt to overcome the challenge of limited data by leveraging the vision foundation model, Segment Anything Model (SAM), prompting it with audio to enhance its ability to segment sounding source objects. While this approach alleviates the model's burden on understanding visual modality by utilizing knowledge of pre-trained SAM, it does not address the fundamental challenge of learning audio-visual correspondence with limited data. To address this limitation, we propose \textbf{AV2T-SAM}, a novel framework that bridges audio features with the text embedding space of pre-trained text-prompted SAM. Our method leverages multimodal correspondence learned from rich text-image paired datasets to enhance audio-visual alignment. Furthermore, we introduce a novel feature, $\mathbf{\textit{\textbf{f}}_{CLIP} \odot \textit{\textbf{f}}_{CLAP}}$, which emphasizes shared semantics of audio and visual modalities while filtering irrelevant noise. Our approach outperforms existing methods on the AVSBench dataset by effectively utilizing pre-trained segmentation models and cross-modal semantic alignment. The source code is released at https://github.com/bok-bok/AV2T-SAM.