MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization

TL;DR

This paper introduces MSRS, a sparse regularization technique enabling training of multimodal speech recognition models from scratch, reducing training time and achieving competitive accuracy without pre-training.

Contribution

MSRS is a novel regularization method that learns sparse structures early in training, allowing models to be trained from scratch with improved efficiency and competitive performance.

Findings

Achieves 21.1% WER on LRS3 benchmark for VSR

Reduces training time by at least 2x

Enables training from scratch by masking vanishing gradients

Abstract

Pre-trained models have been a foundational approach in speech recognition, albeit with associated additional costs. In this study, we propose a regularization technique that facilitates the training of visual and audio-visual speech recognition models (VSR and AVSR) from scratch. This approach, abbreviated as \textbf{MSRS} (Multimodal Speech Recognition from Scratch), introduces a sparse regularization that rapidly learns sparse structures within the dense model at the very beginning of training, which receives healthier gradient flow than the dense equivalent. Once the sparse mask stabilizes, our method allows transitioning to a dense model or keeping a sparse model by updating non-zero values. MSRS achieves competitive results in VSR and AVSR with 21.1% and 0.9% WER on the LRS3 benchmark, while reducing training time by at least 2x. We explore other sparse approaches and show that only MSRS enables training from scratch by implicitly masking the weights affected by vanishing gradients.