Efficient Rehearsal for Continual Learning in ASR via Singular Value Tuning

TL;DR

This paper introduces a novel rehearsal-based continual learning method for ASR that uses Singular Value Decomposition to efficiently adapt models with minimal memory, significantly reducing forgetting.

Contribution

It presents a new SVD-based parameter-efficient rehearsal approach that outperforms existing methods in low-memory scenarios for continual ASR learning.

Findings

Reduces catastrophic forgetting in ASR with minimal rehearsal data.

Outperforms state-of-the-art continual learning methods on multiple benchmarks.

Effective even with only a single utterance per previous task.

Abstract

Continual Learning (CL) in Automatic Speech Recognition (ASR) suffers from catastrophic forgetting when adapting to new tasks, domains, or speakers. A common strategy to mitigate this is to store a subset of past data in memory for rehearsal. However, rehearsal-based methods face key limitations: storing data is often costly, infeasible with pre-trained models, or restricted by privacy regulations. Running existing rehearsal-based methods with smaller memory sizes to alleviate these issues usually leads to degraded performance. We propose a rehearsal-based CL method that remains effective even with minimal memory. It operates in two stages: first, fine-tuning on the new task; second, applying Singular Value Decomposition (SVD) to the changes in linear layers and, in a parameter-efficient manner, retraining only gating vectors on the singular values, which control to extent to which updates from the first stage are accepted, using rehearsal. We extensively test and analyze our method on two monolingual and two multilingual benchmarks. Our method reduces forgetting and outperforms state-of-the-art CL approaches for ASR, even when limited to a single utterance per previous task.