Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networks

TL;DR

This paper introduces a novel language adaptive pre-training method for cross-lingual speech models using sparse sharing sub-networks, significantly improving multilingual speech recognition performance across resource levels.

Contribution

It proposes a sparse sharing sub-network approach for language adaptive training in XLSR models, reducing interference and enhancing performance without manual language-specific components.

Findings

Outperforms baseline XLSR models on multilingual speech recognition

Requires fewer parameters than existing adaptation methods

Effective for both high-resource and low-resource languages

Abstract

Unsupervised cross-lingual speech representation learning (XLSR) has recently shown promising results in speech recognition by leveraging vast amounts of unlabeled data across multiple languages. However, standard XLSR model suffers from language interference problem due to the lack of language specific modeling ability. In this work, we investigate language adaptive training on XLSR models. More importantly, we propose a novel language adaptive pre-training approach based on sparse sharing sub-networks. It makes room for language specific modeling by pruning out unimportant parameters for each language, without requiring any manually designed language specific component. After pruning, each language only maintains a sparse sub-network, while the sub-networks are partially shared with each other. Experimental results on a downstream multilingual speech recognition task show that our proposed method significantly outperforms baseline XLSR models on both high resource and low resource languages. Besides, our proposed method consistently outperforms other adaptation methods and requires fewer parameters.