MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition

TL;DR

This paper introduces MoLE, a multi-lingual speech recognition model that uses language-specific experts and a lightweight tokenizer to improve recognition accuracy across multiple languages, especially low-resource ones.

Contribution

The paper proposes a novel Mixture-of-Language-Expert architecture with a language tokenizer that activates language-specific experts and estimates their reliability for enhanced multi-lingual speech recognition.

Findings

Improved recognition accuracy in multi-lingual scenarios.

Enhanced performance on low-resource languages.

Effective language discrimination and embedding aggregation.

Abstract

Multi-lingual speech recognition aims to distinguish linguistic expressions in different languages and integrate acoustic processing simultaneously. In contrast, current multi-lingual speech recognition research follows a language-aware paradigm, mainly targeted to improve recognition performance rather than discriminate language characteristics. In this paper, we present a multi-lingual speech recognition network named Mixture-of-Language-Expert(MoLE), which digests speech in a variety of languages. Specifically, MoLE analyzes linguistic expression from input speech in arbitrary languages, activating a language-specific expert with a lightweight language tokenizer. The tokenizer not only activates experts, but also estimates the reliability of the activation. Based on the reliability, the activated expert and the language-agnostic expert are aggregated to represent language-conditioned embedding for efficient speech recognition. Our proposed model is evaluated in 5 languages scenario, and the experimental results show that our structure is advantageous on multi-lingual recognition, especially for speech in low-resource language.

Tables

Table 1. Table 1 : Details of train and test datasets

Dataset	En	Es	Fr	Zh	Ja	Total
Train (h)	600	320	400	122	43	1,485
Test (h)	26	26	26	17	6	101

Table 2. Table 2 : Experimental results for CTC-ASR. CER of baseline, and the proposed method. ℒ L ​ R subscript ℒ 𝐿 𝑅 \mathcal{L}_{LR} : language representation loss, LaE : language agnostic expert, Calib. : calibration of LaE.

Model	Lang.	${ℒ}_{LR}$	LaE	Calib.	En	Es	Fr	Zh	Ja	OVR
TFM	Mono				15.20	5.82	8.86	17.82	9.54	10.32
TFM	Multi				18.08	8.33	12.24	19.88	17.65	13.34
MoE	Multi				17.50	7.41	11.20	18.59	15.64	12.43
			✓		15.84	6.48	9.81	17.60	13.56	11.10
MoLE	Multi	✓			17.06	6.94	10.43	17.20	11.07	11.68
			✓		15.79	6.20	9.62	16.84	10.12	10.82
		✓	✓		15.51	6.30	9.27	15.99	9.96	10.59
		✓	✓	✓	15.50	6.10	9.35	16.18	9.26	10.51

Table 3. Table 3 : Experimental results for S2S-ASR

Model	En	Es	Fr	Zh	Ja	OVR
TFM	14.89	5.46	8.63	18.90	15.48	10.37
MoE	13.61	4.84	7.76	17.57	11.01	9.29
MoLE	13.74	5.03	7.74	17.82	10.35	9.37
+Calib.	13.58	4.71	7.56	17.01	8.56	9.05

Equations

$$y=\sum _{i=1}^N{p}_i(x)e_i(x),\text{vspace}-3pt$$

$$y=\sum _{i=1}^N{p}_i(x)e_i(x)\text{mathds}1\{i\in \text{top}k(p(x),k)\},\text{vspace}-3pt$$

$$\begin{array}{r}{\mathcal{L}}_{LR}=-\text{log}p(k=n|z_r)=-\text{log}\frac{e^{cos(z_r,c_n)}}{{\sum }_{n^{\prime }}{e}^{cos(z_r,c_{n^{\prime }})}},\\ \text{where}{c}_n=\frac{1}{|S_n|}\sum _{z_r\in S_n}{z}_r\end{array}\text{vspace}-3pt$$

Full text

**Index Terms— ** Multi-lingual ASR, Mixture of Experts, low-resource ASR

1 Introduction

Humans’ speech and conversation have been written in text, and they have become the basis of the plethora of data these days. Recently, many works have explored linguistic expression in speech signals and converted them into text sequences automatically. Although it had been a difficult task to solve due to acoustic variations, deep learning has accelerated the advances of speech-to-text techniques. Also large-scale speech datasets on the Internet enabled significant performance improvement

In the past, ASR has required phonetic timestamps aligned to speech signals as well as text annotations, and it has consumed a large amount of costs on the annotations. The connectionist temporal classification (CTC) [1] learning method has overcome this problem using an effective forward-backward algorithm. Besides, the advent of learning strategies, there has been large progress in the model view, including sequence-to-sequence models [2, 3, 4] and RNN-transducer models [5, 6, 7].

Although there have been dramatic improvements in ASR, there still remain several challenges. The problem that we address in this paper is the multi-lingual ASR, which distinguishes language-related expression and integrates acoustic processing, simultaneously. Multi-lingual ASR has two major difficulties compared to mono-lingual counterpart. First, if the ASR model cannot detect language well, the model resorts to wrong vocabulary sets, which causes catastrophic results. Many works have avoided this problem by providing explicit language labels, but this solution requires an external language identifier. Second, unbalanced speech data between different languages biases the training to more dominant language with sufficient data while the others are left in an under-fitting state. Several previous works handled these difficulties with various approaches; *e.g. *data efficiency [8, 9], multi-task learning [10, 11], adapter network [12, 13, 14], and meta-learning [15, 16].

In this paper, we propose a novel multi-lingual ASR model, named Mixture-of-Language-Experts (MoLE), which perceives speech in a variety of languages with implicit language router and converts it to text sequences. In MoLE, we leverage a variation of a mixture-of-experts (MoE) network [17, 18, 19, 20] and a switch transformer [21, 22], which have shown great performances in various fields. Our network consists of $N+1$ experts within a MoLE block, including $N$ Language-specific Experts (LsEs) and a Language-agnostic Expert (LaE). We also add a LaE to make up for the confusion of the language gating network and to support generic acoustic modeling. Our proposed model is efficient and suitable for the multi-lingual ASR task since it implicitly distinguishes language types without the help of an external language classifier. Moreover, even if the number of languages increases, the increase in computational complexity in the overall model is extremely minimal because the gating network activates only one LsE per input signal. In experiments, we confirm the superiority of the proposed model compared to both the vanilla transformer model and the MoE-based transformer model. The effectiveness of our model is more pronounced in the results of low-resource language speech recognition.

2 Mixture-of-Experts

Deep MoE [23] has been proposed to increase the representation power of a shallow network architecture, by placing multiple sub-modules or experts in a layer. The experts are dynamically weighted by a gating network, thus it provides sparsity on the network and re-calibrates the network output. The output of a MoE layer is described as,

[TABLE]

where $N$ is the number of experts, $x$ is the input, $p_{i}(\cdot)$ and $e_{i}(\cdot)$ indicate the $i$-th posterior of the gating network and the expert network, respectively. The conventional MoE approaches benefit model scaling and improve performances at the cost of more computations.

Recent works have incorporated the MoE structure with a transformer network [20, 21]. In [21], they proposed a sparse MoE-based transformer. The sparse MoE has solved the increasing computation problem by leveraging only limited number of experts at a time regardless of the total number of experts. The modified gating network selects experts with the highest scores rather than weighting experts which can be described as,

[TABLE]

where $p(\cdot)$ is the posterior vector of the gating network and $\text{top}k(\cdot)$ selects indices of the $k$ highest values. It has reduced significantly computational complexity for the inference, compared to the MoE-based transformer structure; computational cost remains the same regardless of the number of experts. They also have introduced an auxiliary training loss encouraging balanced usage of every expert, to efficiently leverage the network capacity and prevent biased training on a certain expert. The detailed architecture is shown in Fig. 1 (a), where there is an additional general expert for the skip connection. In this paper, we reform the sparse MoE-based transformer suitable for the multi-lingual ASR. We will simply refer to the aforementioned network as MoE in this paper.

3 Proposed Method

We demonstrate our proposed model, Mixture-of-Language-Experts (MoLE), for multi-lingual ASR, with several modifications on the MoE structure to better suit the task. Instead of homogeneous experts in the MoE, the MoLE comprises two types of experts with their different roles: a language-specific expert and a language-agnostic expert. The overall architecture of MoLE is presented in Fig. 1 (b).

3.1 Language-specific Expert (LsE)

We design the language-specific expert which perceives speech signals spoken in a specific language. Thus, we employ $N$ LsEs in a MoLE layer to perceive $N$ languages. The computational complexity of the MoLE does not increase regardless of the number of experts, due to the nature of the sparse MoE structure. The major difference compared to the sparse MoE lies in the gating network. First, our gating network is an LSTM-based model, ingesting the entire input sequence with sequential modeling for a global token. The utterance-wise global token taken from the last frame assigns the input sequence onto a specific expert, while the MoE routes input sequences using framewise tokens, which loses consistency. Since an utterance is usually spoken in one language, it is not appropriate to activate different experts for every input speech frame. Second, instead of the balancing loss in the MoE, we train the gating network using a language representation loss (LRL) inspired in [24]. The language representation loss builds $N$ clusters of speech embeddings and minimizes distances between the centroid of the cluster and belonging embeddings while maximizing distances with the centroids of other languages. The language representation loss $\mathcal{L}_{LR}$ is as follows,

[TABLE]

where $z_{r}$ indicates hidden embedding of a gating network, $c_{n}$ is the centroid of hidden embedding which belongs to $n$-th language, and $S_{n}$ means a subset of speech spoken by $n$-th language. LRL provides guidance on input signals to be assigned to the same expert if they are in the same language. We introduce a metric-based learning strategy to benefit from stable training even in an imbalanced training set, rather than using discriminative training criteria (*e.g. *cross-entropy). Thus, the global tokens are chosen according to the language information in speech signals, and the gating network activates the same expert consistently for speech signals spoken in the same language.

3.2 Language-agnostic Expert (LaE)

In addition to LsEs, the MoLE consists of another expert, named language-agnostic expert (LaE). This expert ingests input data, agnostic to language types in the input speech. The main role of LaE is to analyze acoustic characteristics not related to languages such as speech power, pitch, speaking environments, or speaker identity. The normalization of these features for ASR is necessary, and it should be activated equally regardless of language. Therefore, it is efficient to share an expert for all languages. The other role of LaE is supporting LsEs by re-calibrating the outputs of MoLE. We designed this module to supplement LsEs in case of low confidence $\gamma$ in gating decisions. The highest posterior value $\gamma$ of gating networks is weighted to the outputs of the selected LsE, but the value is significantly variable about input signals (primarily influenced by the confidence of the gating network). Here, we intend to compensate for the low confidence decision of the gating network by relatively increasing the weights of LaE outputs. Thus, we calibrate the output of LaE by $(1-\gamma)$, and it stabilizes the outputs of MoLE. It is particularly advantageous on the multi-lingual ASR since the LaE intervenes more in the acoustic modeling, in case of low confidence on the gating network output.

4 Experiments

4.1 Implementation Details

Datasets The dataset used in experiments is a subset of CommonVoice which contains 5 languages: English (En), Spanish (Es), French (Fr), Chinese (Zh) and Japanese (Ja). The total length of training data and test data are 1,485 hours and 101 hours, respectively, where the amount of each language set is not in balanced. The amount of training and evaluation data of each language is depicted in Tab. 1. In a training mini-batch, we randomly placed speech samples from the entire training set to follow the language distributions of the dataset. The speech signal is transformed to 80-dimensional log-Mel filter-bank energy extracted at every 10ms with 25ms frame length in 16,000Hz sampling rate.

Architecture We implemented 3 CTC-ASR models containing 12 transformer (TFM) blocks, where the block settings are customized according to the model. The baseline model follows similar architecture of [25] with some modifications for a fair comparison. It contains TFM block in every layer, containing a self-attention module and a feed-forward network (FFN) in a TFM block (Fig. 2 (a)). The MoE-based transformer model placed MoE blocks after every TFM block as in Fig. 2 (b). The proposed model has MoLE modules replacing MoE modules, on the 8, 10, and 12th layers, as shown in Fig. 2 (c). Both in MoE and MoLE modules, each expert including LsE and LaE consists of 2 layers of FFN, which is the same as the FFN in the TFM block. For a fair comparison with our model, we placed a common expert similar to LaE, where it is not regulated by the gating networks, in the MoE-based model. The type of gating network in MoLE is LSTM, whereas the MoE block has 2 fully-connected layers as the gating network. Each self-attention layer has 256 dimension with 4 heads, and FFN module has a hidden layer and a projection layer with 2,048 and 256 dimension, respectively. The gating network consists of an LSTM with 64 cells and an fully-connected layer of 5 dimension same as the number of languages. The S2S decoder has 4 TFM layers of 2,048 dim. For the ASR, we prepared the 7,000 word-piece model in total following [26].

4.2 Experimental Results

We compare performances between 3 CTC-ASR models in the character-error-rate (CER). We also analyze the impact of each training strategy through an ablation study.

In Tab. 2, we report the overall ASR performances of baseline models and the proposed model. Compared to the mono-lingual ASR where one model learns a single language, multi-lingual ASR results showed degraded performances in every language since the model confuses vocabulary units that have similar pronunciation across languages. In contrast, MoE and MoLE-based models showed overall improvements compared to the vanilla TFM model. Moreover, the proposed MoLE even surpassed the mono-lingual ASR in Zh and Ja which are low-resource languages, since it lessened the lack of data and confusion of vocabulary by general acoustic modeling and LsE.

Between MoE and MoLE models without the LaE, we found that leveraging LsEs instead of usual experts was significantly effective. The increment in performance was especially noticeable in the case of low-resource languages.

Ablation study We conducted several ablation studies to verify the effectiveness of language representation loss ($\mathcal{L}_{LR}$), language-agnostic expert (LaE), and the calibration on the LaE. In both MoE and MoLE models, introducing LaE reduced the CER with more than 1% in most cases. The LRL brought slight improvements in overall results by stabilizing the gating network for consistent expert activation. Also, the calibration on the LaE outputs with the residual of gating network weights resulted in lower CER, especially in Ja.

4.3 Sequence-to-sequence (S2S)-ASR

To generalize the experimental results for our proposed model, we conducted another experiment, the sequence-to-sequence recognition model. Similar to CTC-ASR experiments, we implemented TFM-based, MoE-based, and MoLE-based transformer in the encoder, respectively and there are 4 layers of MoE-based transformer with a cross-attention in the decoder following to [4]. In Tab. 3, we reported the recognition in CER metric, and the MoE and MoLE approaches showed improved results compared to the TFM. Comparing the MoE and MoLE, we found similar results of CTC-ASR, where there was performance improvement on our proposed model. Specifically, the calibration of LaE led to improvements in the low-resource languages.

5 Conclusion

This work proposed a powerful multi-lingual ASR model, called Mixture-of-Language-Experts (MoLE). We introduced several modifications to the mixture-of-experts approaches, to be favorable to the multi-lingual ASR task. The gating network assigns input sequence to a certain language-specific expert by sentence according to the language type, and the language-agnostic expert perceives it regardless of language. Experiments for CTC-ASR and S2S-ASR verified the superiority of our proposed method with a significant improvement in multi-lingual recognition, especially in low-resource languages. In addition, we conducted several ablation studies to examine our strategies for the MoLE, which led to meaningful recognition improvements.