PACE: Pretrained Audio Continual Learning

TL;DR

This paper introduces PACE, a novel method for audio continual learning with pretrained models, addressing unique challenges like spectral focus and representation drift, and demonstrating significant improvements on multiple benchmarks.

Contribution

The work presents the first systematic benchmark for audio CL with PTMs, analyzes unique challenges, and proposes PACE, a new method combining regularized classifiers and adaptive PEFT for better stability and semantic alignment.

Findings

PACE outperforms state-of-the-art baselines on six audio CL benchmarks.

Analytic classifiers with FSA are promising but face limitations like saturation and drift.

Spectrogram-based perturbations improve stability and reduce representation overlap.

Abstract

Audio is a fundamental modality for analyzing speech, music, and environmental sounds. Although pretrained audio models have significantly advanced audio understanding, they remain fragile in real-world settings where data distributions shift over time. In this work, we present the first systematic benchmark for audio continual learning (CL) with pretrained models (PTMs), together with a comprehensive analysis of its unique challenges. Unlike in vision, where parameter-efficient fine-tuning (PEFT) has proven effective for CL, directly transferring such strategies to audio leads to poor performance. This stems from a fundamental property of audio backbones: they focus on low-level spectral details rather than structured semantics, causing severe upstream-downstream misalignment. Through extensive empirical study, we identify analytic classifiers with first-session adaptation (FSA) as a promising direction, but also reveal two major limitations: representation saturation in coarse-grained scenarios and representation drift in fine-grained scenarios. To address these challenges, we propose PACE, a novel method that enhances FSA via a regularized analytic classifier and enables multi-session adaptation through adaptive subspace-orthogonal PEFT for improved semantic alignment. In addition, we introduce spectrogram-based boundary-aware perturbations to mitigate representation overlap and improve stability. Experiments on six diverse audio CL benchmarks demonstrate that PACE substantially outperforms state-of-the-art baselines, marking an important step toward robust and scalable audio continual learning with PTMs.