Continual Learning for On-Device Speech Recognition using Disentangled Conformers

TL;DR

This paper introduces DisConformer, a novel model architecture with frozen and tunable components, and a continual learning algorithm DisentangledCL, to improve speaker-specific on-device speech recognition while maintaining compute efficiency.

Contribution

It proposes DisConformer with disentangled components and DisentangledCL for efficient continual learning in on-device speech recognition, addressing real-world user adaptation challenges.

Findings

DisConformer outperforms baseline models on LibriSpeech with 15.58% rel. WER reduction.

DisentangledCL significantly improves speaker-specific adaptation, reducing WER by 20.65%.

Models match fully finetuned baselines in some settings.

Abstract

Automatic speech recognition research focuses on training and evaluating on static datasets. Yet, as speech models are increasingly deployed on personal devices, such models encounter user-specific distributional shifts. To simulate this real-world scenario, we introduce LibriContinual, a continual learning benchmark for speaker-specific domain adaptation derived from LibriVox audiobooks, with data corresponding to 118 individual speakers and 6 train splits per speaker of different sizes. Additionally, current speech recognition models and continual learning algorithms are not optimized to be compute-efficient. We adapt a general-purpose training algorithm NetAug for ASR and create a novel Conformer variant called the DisConformer (Disentangled Conformer). This algorithm produces ASR models consisting of a frozen 'core' network for general-purpose use and several tunable 'augment' networks for speaker-specific tuning. Using such models, we propose a novel compute-efficient continual learning algorithm called DisentangledCL. Our experiments show that the DisConformer models significantly outperform baselines on general ASR i.e. LibriSpeech (15.58% rel. WER on test-other). On speaker-specific LibriContinual they significantly outperform trainable-parameter-matched baselines (by 20.65% rel. WER on test) and even match fully finetuned baselines in some settings.