Language-specific Characteristic Assistance for Code-switching Speech Recognition

TL;DR

This paper introduces a language-specific characteristic assistance (LSCA) method for code-switching speech recognition, leveraging language constraints and pre-trained models to improve accuracy without extra shared parameters.

Contribution

The paper proposes LSCA, a novel approach that incorporates language constraints during training and decoding, enhancing code-switching speech recognition performance using pre-trained language-specific models.

Findings

Up to 15.4% relative error reduction on code-switching test set.

Improved performance with training and decoding methods of LSCA.

Effective recognition without extra shared parameters or retraining.

Abstract

Dual-encoder structure successfully utilizes two language-specific encoders (LSEs) for code-switching speech recognition. Because LSEs are initialized by two pre-trained language-specific models (LSMs), the dual-encoder structure can exploit sufficient monolingual data and capture the individual language attributes. However, most existing methods have no language constraints on LSEs and underutilize language-specific knowledge of LSMs. In this paper, we propose a language-specific characteristic assistance (LSCA) method to mitigate the above problems. Specifically, during training, we introduce two language-specific losses as language constraints and generate corresponding language-specific targets for them. During decoding, we take the decoding abilities of LSMs into account by combining the output probabilities of two LSMs and the mixture model to obtain the final predictions. Experiments show that either the training or decoding method of LSCA can improve the model's performance. Furthermore, the best result can obtain up to 15.4% relative error reduction on the code-switching test set by combining the training and decoding methods of LSCA. Moreover, the system can process code-switching speech recognition tasks well without extra shared parameters or even retraining based on two pre-trained LSMs by using our method.