Continual learning using lattice-free MMI for speech recognition

TL;DR

This paper explores regularization-based continual learning methods for neural network acoustic models in speech recognition, demonstrating improved domain adaptation and reduced forgetting using sequence-level LWF with LF-MMI training.

Contribution

It introduces a sequence-level LWF regularization technique that leverages LF-MMI posteriors to enhance continual learning in speech recognition models.

Findings

Sequence-level LWF improves average WER by up to 9.4% relative.

Regular LWF and EWC help mitigate catastrophic forgetting.

The approach effectively adapts models to multiple speech domains.

Abstract

Continual learning (CL), or domain expansion, recently became a popular topic for automatic speech recognition (ASR) acoustic modeling because practical systems have to be updated frequently in order to work robustly on types of speech not observed during initial training. While sequential adaptation allows tuning a system to a new domain, it may result in performance degradation on the old domains due to catastrophic forgetting. In this work we explore regularization-based CL for neural network acoustic models trained with the lattice-free maximum mutual information (LF-MMI) criterion. We simulate domain expansion by incrementally adapting the acoustic model on different public datasets that include several accents and speaking styles. We investigate two well-known CL techniques, elastic weight consolidation (EWC) and learning without forgetting (LWF), which aim to reduce forgetting by preserving model weights or network outputs. We additionally introduce a sequence-level LWF regularization, which exploits posteriors from the denominator graph of LF-MMI to further reduce forgetting. Empirical results show that the proposed sequence-level LWF can improve the best average word error rate across all domains by up to 9.4% relative compared with using regular LWF.