Batch-normalized joint training for DNN-based distant speech recognition

TL;DR

This paper introduces a batch-normalized joint training method for deep neural networks in distant speech recognition, improving robustness in adverse acoustic conditions by better aligning speech enhancement and recognition modules.

Contribution

It proposes a fully batch-normalized architecture for joint training of speech enhancement and recognition DNNs, addressing non-stationary output distributions during optimization.

Findings

Significantly outperforms other methods in challenging environments

Improves robustness of distant speech recognition systems

Effective across various datasets and acoustic conditions

Abstract

Improving distant speech recognition is a crucial step towards flexible human-machine interfaces. Current technology, however, still exhibits a lack of robustness, especially when adverse acoustic conditions are met. Despite the significant progress made in the last years on both speech enhancement and speech recognition, one potential limitation of state-of-the-art technology lies in composing modules that are not well matched because they are not trained jointly. To address this concern, a promising approach consists in concatenating a speech enhancement and a speech recognition deep neural network and to jointly update their parameters as if they were within a single bigger network. Unfortunately, joint training can be difficult because the output distribution of the speech enhancement system may change substantially during the optimization procedure. The speech recognition module would have to deal with an input distribution that is non-stationary and unnormalized. To mitigate this issue, we propose a joint training approach based on a fully batch-normalized architecture. Experiments, conducted using different datasets, tasks and acoustic conditions, revealed that the proposed framework significantly overtakes other competitive solutions, especially in challenging environments.

Figures

Tables

Table 1. Table 1 : Performance of the proposed joint training approach compared with other competitive DNN-based systems.

	TIMIT	WSJ	WSJ
	Rev	Rev	Rev+Noise
Single big DNN	31.5	8.1	14.3
SE + clean SR	31.1	8.5	15.7
SE + matched SR	30.1	8.0	13.7
SE + SR joint training	29.2	7.8	12.7

Table 2. Table 2 : Analysis of the role played by batch normalization within the proposed joint training framework.

	Without Pre-Training		With Pre-Training
	no-BN	with-BN	no-BN	with-BN
TIMIT-Rev	34.2	29.2	32.6	29.5
WSJ-Rev	9.0	7.8	8.8	7.8
WSJ-Rev+Noise	15.7	12.7	15.0	12.9

Equations

$${\theta }_{SE}\leftarrow {\theta }_{SE}-lr\ast (g_{SE}+\lambda g_{SR}).$$

Full text

To address this concern, a promising approach consists in concatenating a speech enhancement and a speech recognition deep neural network and to jointly update their parameters as if they were within a single bigger network. Unfortunately, joint training can be difficult because the output distribution of the speech enhancement system may change substantially during the optimization procedure. The speech recognition module would have to deal with an input distribution that is non-stationary and unnormalized. To mitigate this issue, we propose a joint training approach based on a fully batch-normalized architecture.

Experiments, conducted using different datasets, tasks and acoustic conditions, revealed that the proposed framework significantly overtakes other competitive solutions, especially in challenging environments.

**Index Terms— ** speech recognition, speech enhancement, joint training, deep neural networks.

1 Introduction

Automatic Speech Recognition (ASR) [1], thanks to the substantial performance improvement achieved with modern deep learning technologies [2], has recently been applied in several fields, and it is currently used by millions of users worldwide. Nevertheless, most state-of-the-art systems are still based on close-talking solutions, forcing the user to speak very close to a microphone-equipped device. It is easy to predict, however, that in the future users will prefer to relax the constraint of handling or wearing any device to access speech recognition services, requiring technologies able to cope with a distant-talking (far-field) interaction.

In the last decade, several efforts have been devoted to improving Distant Speech Recognition (DSR) systems. Valuable examples include the AMI/AMIDA projects [3], who were focused on automatic meeting transcription, DICIT [4] which investigated voice-enabled TVs and, more recently, DIRHA which addressed speech-based domestic control. The progress in the field was also fostered by the considerable success of some international challenges such as CHiME [5, 6] and REVERB [7].

Despite the great progress made in the past years, current systems still exhibit a significant lack of robustness to acoustic conditions characterized by non-stationary noises and acoustic reverberation [8]. To counteract such adversities, even the most recent DSR systems [9] must rely on a combination of several interconnected technologies, including for instance speech enhancement [10], speech separation [11], acoustic event detection and classification [12, 13], speaker identification [14], speaker localization [15, 16], just to name a few.

A potential limitation of most current solutions lies in the weak matching and communication between the various modules being combined. For example, speech enhancement and speech recognition are often designed independently and, in several cases, the enhancement system is tuned according to metrics which are not directly correlated with the final ASR performance.

An early attempt to mitigate this issue was published in [17]. In LIMABEAM, the goal was to tune the parameters of a microphone array beamformer by maximizing the likelihood obtained through a GMM-based speech recognizer. Another approach was proposed in [18], where a front-end for feature extraction and a GMM-HMM back-end were jointly trained using maximum mutual information.

An effective integration between the various systems, however, was very difficult for many years, mainly due to the different nature of the technologies involved at the various steps. Nevertheless, the recent success of deep learning has not only largely contributed to the substantial improvement of the speech recognition part of a DSR system [19, 20, 21, 22, 23, 24, 25], but has also enabled the development of competitive DNN-based speech enhancement solutions [26, 27, 28]. Within the DNN framework, one way to achieve a fruitful integration of the various components is joint training. The core idea is to pipeline a speech enhancement and a speech recognition deep neural networks and to jointly update their parameters as if they were within a single bigger network. Although joint training for speech recognition is still an under-explored research direction, such a paradigm is progressively gaining more attention and some interesting works in the field have been recently published [29, 30, 31, 32, 33, 34, 35].

In this paper, we contribute to this line of research by proposing an approach based on joint training of a speech enhancement and a speech recognition DNN coupled with batch normalization in order to help making one network less sensitive to changes in the other. Batch normalization [36], which has recently been proposed in the machine learning community, has been shown crucial to significantly improve both the convergence and the performance of the proposed joint training algorithm. Differently to previous works [30, 31], thanks to batch normalization, we are able to effectively train the joint architecture even without any pre-training steps. Another interesting aspect concerns a deeper study of a gradient weighting strategy, which ended up being particularly effective to improve performance.

The experimental validation has been carried out in a distant-talking scenario considering different training datasets, tasks and acoustic conditions.

2 Batch-normalized joint training

The proposed architecture is depicted in Fig. 1. A bigger joint DNN is built by concatenating a speech enhancement and a speech recognition MLP. The speech enhancement DNN is fed with the noisy features $x_{noise}$ gathered within a context window and tries to reconstruct at the output the original clean speech (regression task). The speech recognition DNN is fed by the enhanced features $x_{enh}$ estimated at the previous layer and performs phone predictions $y_{pred}$ at each frame (classification task). The architecture of Fig. 1 is trained with the algorithm described in Alg. 1.

The basic idea is to perform a forward pass, compute the loss functions at the output of each DNN (mean-squared error for speech enhancement and negative multinomial log-likelihood for speech recognition), compute and weight the corresponding gradients, and back-propagate them. In the joint training framework, the speech recognition gradient is also back-propagated through the speech enhancement DNN. Therefore, at the speech enhancement level, the parameter updates not only depend on the speech enhancement cost function but also on the speech recognition loss, as shown by Eq. 1:

[TABLE]

In Eq. 1, $\theta_{SE}$ are the parameters of the speech enhancement DNN, $g_{SE}$ is the gradient of such parameters computed from the speech enhancement cost function (mean squared error), while $g_{SR}$ is the gradient of $\theta_{SE}$ computed from the speech recognition cost function (multinomial log-likelihood). Finally, $\lambda$ is a hyperparameter for weighting $g_{SR}$ and $lr$ is the learning rate.

The key intuition behind joint training is that since the enhancement process is in part guided by the speech recognition cost function, the front-end would hopefully be able to provide enhanced speech which is more suitable and discriminative for the subsequent speech recognition task.

From a machine learning perspective, this solution can also be considered as a way of injecting a useful task-specific prior knowledge into a deep neural network. On the other hand, it is well known that training deep architectures is easier when some hints are given about the targeted function [37]. As shown previously [37], such prior knowledge becomes progressively more precious as the complexity of the problem increases and can thus be very helpful for a distant speech recognition task. Similarly to the current work, in [37, 38] a task-specific prior knowledge has been injected into an intermediate layer of a DNN for better addressing an image classification problem.

In our case, we exploit the prior assumption that to solve our specific problem, it is reasonable to first enhance the features and, only after that, perform the phone classification. Note that this is certainly not the only way of solving the problem, but among all the possible functions able to fit the training data, we force the system to choose from a more restricted subset, potentially making training easier. On the other hand, good prior knowledge is helpful to defeat the curse of dimensionality, and a complementary view is thus to consider the proposed joint training as a regularizer. According to this vision, the weighting parameter $\lambda$ of Eq. 1 can be regarded as a regularization hyperparameter, as will be better discussed in Sec. 4.4.

2.1 Batch normalization

Training DNNs is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This problem, known as internal covariate shift, slows down the training of deep neural networks. Batch normalization [36], which has been recently proposed in the machine learning community, addresses this issue by normalizing the mean and the variance of each layer for each training mini-batch, and back-propagating through the normalization step. It has been long known that the network training converges faster if its inputs are properly normalized [39] and, in such a way, batch normalization extends the normalization to all the layers of the architecture. However, since a per-layer normalization may impair the model capacity, a trainable scaling parameter $\gamma$ and a trainable shifting parameter $\beta$ are introduced in each layer to restore the representational power of the network.

The idea of using batch normalization for the joint training setup is motivated by a better management of the internal covariate shift problem, which might be crucial when training our (very) deep joint architecture. As will be shown in Sec. 1, batch normalization allows us to significantly improve the performance of the system, to speed-up the training, and to avoid any time-consuming pre-training steps.

Particular attention should anyway be devoted to the initialization of the $\gamma$ parameter. Contrary to [36], where it was initialized to unit variance ($\gamma=1$), in this work we have observed better performance and convergence properties with a smaller variance initialization ($\gamma=0.1$). A similar outcome has been found in [40], where fewer vanishing gradient problems are empirically observed with small values of $\gamma$ in the case of recurrent neural networks.

2.2 System details

The features considered in this work are standard 39 Mel-Cepstral Coefficients (MFCCs) computed every 10 ms with a frame length of 25 ms. The speech enhancement DNN is fed with a context of 21 consecutive frames and predicts (every 10 ms) 11 consecutive frames of enhanced MFCC features. The idea of predicting multiple enhanced frames was also explored in [31]. All the layers used Rectified Linear Units (ReLU), except for the output of the speech enhancement (linear) and the output of speech recognition (softmax). Batch normalization [36] is employed for all the hidden layers, while dropout [41] is adopted in all part of the architecture, except for the output layers.

The datasets used for joint training are obtained through a contamination of clean corpora (i.e., TIMIT and WSJ) with noise and reverberation. The labels for the speech enhancement DNN (denoted as $x_{clean}$ in Alg.1) are the MFCC features of the original clean datasets. The labels for the speech recognition DNN (denoted as $y_{lab}$ in Alg.1) are derived by performing a forced alignment procedure on the original training datasets. See the standard s5 recipe of Kaldi for more details [42].

The weights of the network are initialized according to the Glorot initialization [43], while biases are initialized to zero. Training is based on a standard Stochastic Gradient Descend (SGD) optimization with mini-batches of size 128. The performance on the development set is monitored after each epoch and the learning rate is halved when the performance improvement is below a certain threshold. The training ends when no significant improvements have been observed for more than four consecutive epochs. The main hyperparameters of the system (i.e., learning rate, number of hidden layers, hidden neurons per layer, dropout factor and $\lambda$) have been optimized on the development set.

The proposed system, which has been implemented with Theano [44], has been coupled with the Kaldi toolkit [42] to form a context-dependent DNN-HMM speech recognizer.

2.3 Relation to prior work

Similarly to this paper, a joint training framework has been explored in [29, 30, 31, 32, 33, 34, 35]. A key difference with previous works is that we propose to combine joint training with batch normalization. In [30, 31], for instance, the joint training was actually performed as a fine-tuning procedure, which was carried out only after training the two networks independently. A critical aspect of such an approach is that the learning rate adopted in the fine-tuning step has to be properly selected in order to really take advantage of pre-training. With batch normalization we are able not only to significantly improve the performance of the system, but also to perform joint training from scratch, skipping any pre-training phase.

Another interesting aspect of this work is a deeper study of the role played by the gradient weighting factor $\lambda$.

3 Corpora and tasks

In order to provide an accurate evaluation of the proposed technique, the experimental validation has been conducted using different training datasets, different tasks and various environmental conditions111To allow reproducibility of the results reported in this paper, the code of our joint-training system will be available at https://github.com/mravanelli. In the same repository, all the scripts needed for the data contamination will be available. The public distribution of the DIRHA-English dataset is under discussion with the Linguistic Data Consortium (LDC)..

The experiments with TIMIT are based on a phoneme recognition task (aligned with the Kaldi s5 recipe). The original training dataset has been contaminated with a set of realistic impulse responses measured in a real apartment. The reverberation time ($T_{60}$) of the considered room is about 0.7 seconds. Development and test data have been simulated with the same approach. More details about the data contamination approach can be found in [45, 46, 47].

The WSJ experiments are based on the popular wsj5k task (aligned with the CHiME 3 [6] task) and are conducted under two different acoustic conditions. For the WSJ-Rev case, the training set is contaminated with the same set of impulse responses adopted for TIMIT. For the WSJ-Rev+Noise case, we also added non-stationary noises recorded in a domestic context (the average SNR is about 10 dB). The test phase is carried out with the DIRHA English Dataset, consisting of 409 WSJ sentences uttered by six native American speakers in the above mentioned apartment. For more details see [48, 47].

4 Experiments

4.1 Close-talking baselines

The Phoneme Error Rate (PER%) obtained by decoding the original test sentences of TIMIT is $19.5\%$ (using DNN models trained with the original dataset). The Word Error Rate (WER%) obtained by decoding the close-talking WSJ sentences is $3.3\%$. It is worth noting that, under such favorable acoustic conditions, the DNN model leads to a very accurate sentence transcription, especially when coupled with a language model.

4.2 Joint training performance

In Table 1, the proposed joint training approach is compared with other competitive strategies.

In particular, the first line reports the results obtained with a single neural network. The size of the network has been optimized on the development set (4 hidden layers of 1024 neurons for TIMIT, 6 hidden layers of 2048 neurons for WSJ cases). The second line shows the performance obtained when the speech enhancement neural network (4 hidden layers of 2048 neurons for TIMIT, 6 hidden layers of 2048 neurons for WSJ) is trained independently and later coupled with the close-talking DNN of Sec. 4.1. These results are particularly critical because, especially in adverse acoustic conditions, the speech enhancement model introduces significant distortions that a close-talking DNN trained in the usual ways is not able to cope with. To partially recover such a critical mismatch, one approach is to first train the speech enhancement, then pass all the training features though the speech enhancement DNN, and, lastly, train the speech recognition DNN with the dataset processed by the speech enhancement. The third line shows results obtained with such a matched training approach. The last line reports the performance achieved with the proposed joint training approach. Batch normalization is adopted for all the systems considered in Table 1.

Although joint training exhibits in all the cases the best performance, it is clear that such a technique is particularly helpful especially when challenging acoustic conditions are met. For instance, a relative improvement of about $8\%$ over the most competitive matched training system is obtained for the WSJ task in noisy and reverberant conditions.

4.3 Role of batch normalization

In Table 2, the impact of batch normalization on the joint training framework is shown. The first two columns report, respectively, the results obtained with and without batch normalization when no pre-training techniques are employed. The impact of pre-training is studied in the last two columns. The pre-training strategy considered here consists of initializing the two DNNs with the matched training system discussed in Sec. 4.2, and performing a fine-tuning phase with a reduced learning rate. The column corresponding to the pre-training without batch normalization represents a system that most closely matches the approaches followed in [30, 31].

Table 2 clearly shows that batch normalization is particularly helpful. For instance, a relative improvement of about 23% is achieved when batch normalization is adopted for the WSJ task in a noisy and reverberant scenario. The key importance of batch normalization is also highlighted in Fig. 2, where the evolution during training of the frame-level phone error rate (for the TIMIT-Rev dataset) is reported with and without batch normalization. From the figure it is clear that batch normalization, when applied to the considered deep joint architecture, ensures a faster convergence and a significantly better performance. Moreover, as shown in Table 2, batch normalization eliminates the need of DNN pre-training, since similar (or even slightly worse results) are obtained when pre-training and batch normalization are used simultaneously.

4.4 Role of the gradient weighting

In Fig. 3, the role of the gradient weighting factor $\lambda$ is highlighted.

From the figure one can observe that small values of $\lambda$ lead to a situation close to underfitting, while higher values of $\lambda$ cause overfitting. The latter result is somewhat expected since, intuitively, with very large values of $\lambda$ the speech enhancement information tends to be neglected and training relies on the speech recognition gradient only.

In the present work, we have seen that values of $\lambda$ ranging from 0.03 to 0.1 provide the best performance. Note that these values are smaller than that considered in [30, 29], where a pure gradient summation ($\lambda=1$) was adopted. We argue that this result is due to the fact that, as observed in [40], the norm of the gradient decays very slowly when adopting batch normalization with a proper initialization of $\gamma$, even after the gradient has passed through many hidden layers. This causes the gradient backpropagated through the speech recognition network and into the speech enhancement network to be very large.

5 Conclusion

In this paper, a novel approach for joint training coupled with batch normalization is proposed. The experimental validation, conducted considering different tasks, datasets and acoustic conditions, showed that batch-normalized joint training is particularly effective in challenging acoustic environments, characterized by both noise and reverberation. In particular, batch normalization was of crucial importance for improving the system performance. A remarkable result is the relative improvement of about 23% obtained for the WSJ task in a noisy and reverberant scenario when batch normalization is used within the joint training framework.

This system can be seen as a first step towards a better and more fruitful integration of the various technologies involved in current distant speech recognition systems. Future efforts for improving the current solution will be devoted to progressively involve different NN architectures or to embed other technologies such as speech separation, speaker identification and acoustic scene analysis.