SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation

TL;DR

SeaVIS is an innovative online framework for audio-visual instance segmentation that effectively associates instances across continuous video streams by integrating audio cues and visual features, enabling real-time processing and improved sound-based object tracking.

Contribution

The paper introduces SeaVIS, the first online AVIS framework utilizing CCAF and AGCL to improve instance association and sound activity encoding in real-time videos.

Findings

Outperforms existing models on AVISeg dataset

Achieves real-time inference speed

Enhances sound-following accuracy

Abstract

Recently, an audio-visual instance segmentation (AVIS) task has been introduced, aiming to identify, segment and track individual sounding instances in videos. However, prevailing methods primarily adopt the offline paradigm, that cannot associate detected instances across consecutive clips, making them unsuitable for real-world scenarios that involve continuous video streams. To address this limitation, we introduce SeaVIS, the first online framework designed for audio-visual instance segmentation. SeaVIS leverages the Causal Cross Attention Fusion (CCAF) module to enable efficient online processing, which integrates visual features from the current frame with the entire audio history under strict causal constraints. A major challenge for conventional VIS methods is that appearance-based instance association fails to distinguish between an object's sounding and silent states, resulting in the incorrect segmentation of silent objects. To tackle this, we employ an Audio-Guided Contrastive Learning (AGCL) strategy to generate instance prototypes that encode not only visual appearance but also sounding activity. In this way, instances preserved during per-frame prediction that do not emit sound can be effectively suppressed during instance association process, thereby significantly enhancing the audio-following capability of SeaVIS. Extensive experiments conducted on the AVISeg dataset demonstrate that SeaVIS surpasses existing state-of-the-art models across multiple evaluation metrics while maintaining a competitive inference speed suitable for real-time processing.