Continual Speech Learning with Fused Speech Features

TL;DR

This paper proposes a continual learning framework for speech models using a gated-fusion layer on Whisper, enabling dynamic task-specific feature selection and significantly improving adaptation across multiple speech tasks.

Contribution

It introduces a novel continual speech learning setup with a gated-fusion layer on Whisper, enhancing task adaptation without full retraining.

Findings

Significant accuracy improvements over traditional methods

Effective adaptation to new speech tasks

Demonstrated across six speech processing tasks

Abstract

Rapid growth in speech data demands adaptive models, as traditional static methods fail to keep pace with dynamic and diverse speech information. We introduce continuous speech learning, a new set-up targeting at bridging the adaptation gap in current speech models. We use the encoder-decoder Whisper model to standardize speech tasks into a generative format. We integrate a learnable gated-fusion layer on the top of the encoder to dynamically select task-specific features for downstream tasks. Our approach improves accuracy significantly over traditional methods in six speech processing tasks, demonstrating gains in adapting to new speech tasks without full retraining.