Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation

TL;DR

This paper introduces a new benchmark and baseline methods for continual learning in audio-visual segmentation, addressing the challenge of dynamic environments and preventing catastrophic forgetting.

Contribution

It presents the first exemplar-free continual learning benchmark for AVS and proposes ATLAS with Low-Rank Anchoring to improve lifelong audio-visual perception.

Findings

ATLAS achieves competitive performance in continual AVS scenarios

The benchmark enables evaluation of models in dynamic, real-world environments

LRA effectively mitigates catastrophic forgetting in AVS models

Abstract

Audio-Visual Segmentation (AVS) aims to produce pixel-level masks of sound producing objects in videos, by jointly learning from audio and visual signals. However, real-world environments are inherently dynamic, causing audio and visual distributions to evolve over time, which challenge existing AVS systems that assume static training settings. To address this gap, we introduce the first exemplar-free continual learning benchmark for Audio-Visual Segmentation, comprising four learning protocols across single-source and multi-source AVS datasets. We further propose a strong baseline, ATLAS, which uses audio-guided pre-fusion conditioning to modulate visual feature channels via projected audio context before cross-modal attention. Finally, we mitigate catastrophic forgetting by introducing Low-Rank Anchoring (LRA), which stabilizes adapted weights based on loss sensitivity. Extensive experiments demonstrate competitive performance across diverse continual scenarios, establishing a foundation for lifelong audio-visual perception. Code is available at${}^{*}$\footnote{Paper under review} - \hyperlink{https://gitlab.com/viper-purdue/atlas}{https://gitlab.com/viper-purdue/atlas} \keywords{Continual Learning \and Audio-Visual Segmentation \and Multi-Modal Learning}