Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models

TL;DR

This paper investigates catastrophic forgetting in end-to-end spoken language models and evaluates mitigation strategies, finding experience replay most effective for balancing knowledge retention and new learning.

Contribution

It introduces and compares three mitigation strategies for catastrophic forgetting in spoken language models, highlighting the effectiveness of experience replay.

Findings

Experience replay significantly reduces forgetting.

Combining mitigation strategies yields better results.

Insights for more robust SLM training pipelines.

Abstract

End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS and spoken question answering (SQA). Although this multi-stage continual learning equips LLMs with both speech understanding and generation capabilities, the substantial differences in task and data distributions across stages can lead to catastrophic forgetting, where previously acquired knowledge is lost. This paper investigates catastrophic forgetting and evaluates three mitigation strategies-model merging, discounting the LoRA scaling factor, and experience replay to balance knowledge retention with new learning. Results show that experience replay is the most effective, with further gains achieved by combining it with other methods. These findings provide insights for developing more robust and efficient SLM training pipelines.