Towards Adapting NMF Dictionaries Using Total Variability Modeling for Noise-Robust Acoustic Features

TL;DR

This paper introduces a novel noise-robust acoustic feature extraction method that adapts NMF dictionaries using Total Variability Modeling, without requiring parallel clean-noisy speech data, and demonstrates competitive performance on noisy speech recognition tasks.

Contribution

The paper presents a new algorithm combining Total Variability Modeling with NMF for utterance-specific noise adaptation, avoiding the need for parallel training data.

Findings

Features perform comparably to baseline features on noisy data.

Proposed features are robust to unseen noise conditions.

Method does not require parallel clean-noisy speech corpus.

Abstract

We propose an algorithm to extract noise-robust acoustic features from noisy speech. We use Total Variability Modeling in combination with Non-negative Matrix Factorization (NMF) to learn a total variability subspace and adapt NMF dictionaries for each utterance. Unlike several other approaches for extracting noise-robust features, our algorithm does not require a training corpus of parallel clean and noisy speech. Furthermore, the proposed features are produced by an utterance-specific transform, allowing the features to be robust to the noise occurring in each utterance. Preliminary results on the Aurora 4 + DEMAND noise corpus show that our proposed features perform comparably to baseline acoustic features, including features calculated from a convolutive NMF (CNMF) model. Moreover, on unseen noises, our proposed features gives the most similar word error rate to clean speech compared to the baseline features.

Tables

Table 1. Table 1: Connections between NMF matrices and TVM inputs.

TVM Inputs	NMF Matrices
Features	Magnitude spectrogram
GMM mean supervector	Vectorized NMF dictionary
Cluster posteriors	Normalized activation matrix

Table 2. Table 2: WER for the Aurora 4 + DEMAND test set in the three noise conditions.

Condition	MFCC	PNCC	CNMF	Proposed
clean	$3.68$	$3.96$	$3.40$	$5.60$
noisy, seen	$6.05$	$6.71$	$6.04$	$8.53$
noisy, unseen	$4.27$	$4.64$	$4.38$	$5.95$
Average	$5.28$	$5.81$	$5.28$	$7.46$

Equations

$$D_{GKL}\left(\mathbf{V}\Vert \hat{\mathbf{V}}\right)=\sum _{i=1}^d\sum _{j=1}^t{\mathbf{V}}_{ij}\ln \left(\frac{{\mathbf{V}}_{ij}}{{\hat{\mathbf{V}}}_{ij}}\right)-{\mathbf{V}}_{ij}+{\hat{\mathbf{V}}}_{ij},$$

$${\mathbf{q}}_u\sim \mathcal{N}(0,\mathbf{I})$$

$${\mathbf{M}}_u={\mathbf{M}}_0+\mathbf{T}{\mathbf{q}}_u$$

$$C=D_{GKL}\left({\mathbf{V}}_{\text{all}}\Vert {\hat{\mathbf{V}}}_{\text{all}}\right)+\lambda \sum _{i=1}^k\sum _{j=1}^t{\mathbf{H}}_{\text{ubm}(i,j)},$$

$${\mathbf{W}}_{\text{ubm}}$$

$${\mathbf{H}}_{\text{ubm}}$$

$${\mathbf{H}}_u\leftarrow {\mathbf{H}}_u\odot \frac{{\mathbf{W}}_{\text{ubm}}^{\mathsf{T}}\frac{{\mathbf{V}}_u}{{\hat{\mathbf{V}}}_u}}{{\mathbf{W}}_{\text{ubm}}^{\mathsf{T}}{1}_{d\times t}+\lambda },$$

$${\mathbf{n}}_u$$

$${\mathbf{F}}_u$$

$${\bar{\mathbf{F}}}_{u\left(:,c\right)}$$

$$\begin{gathered} {\mathbf{\Sigma }}_c=\frac{1}{n_{uc}}\sum _{u=1}^U{\tilde{\mathbf{H}}}_{u(c,t)}\left(\log \left({\mathbf{V}}_{u(:,t)}\right)-\log \left({\mathbf{W}}_{\text{ubm}(:,c)}\right)\right) \\ {\left(\log \left({\mathbf{V}}_{u(:,t)}\right)-\log \left({\mathbf{W}}_{\text{ubm}(:,c)}\right)\right)}^{\mathsf{T}}, \end{gathered}$$

$${\mathbf{\mu }}_{{\mathbf{q}}_u}$$

$${\mathbf{\Sigma }}_{{\mathbf{q}}_u}$$

$${\mathbf{T}}_c\leftarrow \left(\sum _{u=1}^U{\bar{\mathbf{F}}}_{u(:,c)}{\mathbf{\mu }}_{{\mathbf{q}}_u}^{\mathsf{T}}\right){\left(\sum _{u=1}^U{n}_{uc}\left({\mathbf{\Sigma }}_{{\mathbf{q}}_u}+{\mathbf{\mu }}_{{\mathbf{q}}_u}{\mathbf{\mu }}_{{\mathbf{q}}_u}^{\mathsf{T}}\right)\right)}^{-1}$$

$${\mathbf{w}}_a=\exp \left(\log \left({\mathbf{w}}_{\text{ubm}}\right)+\mathbf{T}{\mathbf{q}}_a\right)$$

Full text

Index Terms: Automatic speech recognition, ivectors, NMF, total variability modeling

1 Introduction

Automatic speech recognition (ASR) systems are being increasingly deployed on a wide range of devices for a wide range of applications. Speech offers a natural and efficient way to interact with these devices. Furthermore, speech contains paralinguistic content that devices can use to modify their outputs or behavior. For example, ASR systems in call centers can use the emotion of customers to better serve them or mitigate conflicts [1]. Given the wide usage scenarios, ASR systems need to perform robustly in different acoustic environments, with various background noises and channel conditions, and reliably recognize speech with different dialects and accents. Thus, there has been increasing research in making ASR systems more robust to various real-world conditions. Some techniques researchers have developed include speech denoising [2], feature enhancement [3], feature transformation [4], and acoustic model adaptation [5, 6].

Speech denoising is one straightforward way to make ASRs robust to background noise. Pre-processing the speech with a noise-removal algorithm reduces the mismatch between features extracted at test time compared to the features used to train the ASR. Common speech denoising algorithms include Weiner filtering and spectral subtraction [7]. Non-negative matrix factorization (NMF) [8, 9] is also widely used for denoising. The drawback with speech denoising is that it usually introduces distortion and artifacts, such as musical noise, and has been shown to degrade ASR performance [10, 11]. Moreover, the artifacts are usually amplified when the background noise is highly non-stationary or energetic.

To overcome the drawbacks of speech denoising, researchers have investigated extracting acoustic features directly from noisy speech that are robust to noise. Moreno et al. introduced Vector Taylor Series (VTS) features [12], which uses the Taylor series expansion of the noisy signal to model the effect of noise and channel characteristics on the speech statistics. Deng et al. proposed the Stereo-based Piecewise Linear Compensation for Environments (SPLICE) algorithm [13] for generating noise-robust features by assuming that clean speech cepstral vectors have a piece-wise linear relationship to noisy speech cepstral vectors. Power-Normalized Cepstral Coefficients (PNCC) [14] draw inspiration from human auditory processing for generating noise-robust features, and were shown to reduce word error rates on noisy speech compared to Mel-Frequency Cepstral Coefficients (MFCC) and Relative Spectral Perceptual Linear Prediction (RASTA-PLP) coefficients. Recently, an NMF-based approach was proposed [15], where speech and noise dictionaries are trained on clean and noisy speech, and the coefficients in terms of these dictionaries are used as acoustic features. They showed that the NMF-based approach gives better ASR performance than log-mel features or denoising the speech.

In this work, we expand upon the NMF-based approach. On a training set, we learn a universal background model (UBM) dictionary, and then use total variability modeling [16] to learn a subspace for adapting the UBM dictionary to different noise and channel conditions. The advantages of this proposed method over the one in [15] are two-fold:

The training set does not require parallel clean and noisy utterances, and 2. 2.

The dictionary can be adapted for each utterance at test time, allowing for better modeling of the acoustic conditions in each utterance.

In the following sections, we provide a brief overview of NMF and total variability modeling, followed by our proposed noise-robust acoustic feature algorithm. Section 4 describes our experiments and offers insights into the results, and Section 5 offers our concluding remarks and future directions.

2 Background

2.1 Non-negative Matrix Factorization

NMF decomposes a non-negative matrix $\mathbf{V}\in\mathbb{R}_{+}^{d\times t}$ into the product of a non-negative dictionary $\mathbf{W}\in\mathbb{R}_{+}^{d\times k}$ and non-negative activation matrix $\mathbf{H}\in\mathbb{R}_{+}^{k\times t}$. Because of the non-negative constraint, the decomposition is purely additive, and one can think of the dictionary as containing $k$ components that are added together by the activation matrix to approximate the input matrix. In the case of speech, the input matrix is typically the magnitude spectrogram, and the dictionary contains spectral “building blocks” required to reconstruct the spectrogram.

For speech processing, it has been shown that the generalized KL divergence gives slightly better performance as the NMF cost function compared to the squared Euclidean distance [17]. Defining $\hat{\mathbf{V}}=\mathbf{W}\mathbf{H}$, the generalized KL divergence between $\mathbf{V}$ and $\hat{\mathbf{V}}$ is

[TABLE]

where $V_{ij}$ refers to the element in row $i$ and column $j$ of $\mathbf{V}$. Lee and Seung derived iterative multiplicative updates for $\mathbf{W}$ and $\mathbf{H}$ to minimize Equation 1 [9]. The advantages of using multiplicative updates over standard gradient descent updates are that no step size parameter is required and $\mathbf{W}$ and $\mathbf{H}$ are guaranteed to stay non-negative at each iteration if they are initialized with non-negative values. The NMF decomposition for the case of speech can be visualized in Figure 1.

2.2 Total Variability Modeling

The Total Variability Model (TVM) [16] is a tool which can be used to capture distributional differences between sequences of feature vectors within a fixed dimensional representation. In particular, the assumption is that the feature vectors follow a distribution which has the form of a Gaussian Mixture Model (GMM) where the mean vectors corresponding to different Gaussian components vary across different utterances (in a constrained manner).

Let $\mathbf{V}=\left\{\mathbf{V}_{u}\right\}_{u=1}^{U}$ be the collection of acoustic feature vectors in a dataset comprising $U$ utterances, where $\mathbf{V}_{u}=\left\{\bm{{v}}_{u\tau}\right\}_{\tau=1}^{t_{u}}$ denotes the feature vector sequence of length $t_{u}$ from a specific utterance $u$. Let $d$ be the dimensionality of each feature vector: $\bm{{v}}_{ut}\in\mathbb{R}^{d}$.

It is assumed that with every utterance $u$, there is an associated vector $\bm{{q}}_{u}\in\mathbb{R}^{s}$, known as the ivector for that utterance, such that the conditional distribution of $\bm{{v}}_{u\tau}$ given $\bm{{q}}_{u}$ is a GMM with $k$ components, and parameters $\left\{p_{c},\bm{{\mu}}_{uc}=\bm{{\mu}}_{c}+\mathbf{T}_{c}\bm{{q}}_{u},\mathbf{\Sigma}_{c}\right\}_{c=1}^{k}$ where $p_{c}\in\mathbb{R},\bm{{\mu}}_{c}\in\mathbb{R}^{d},\mathbf{T}_{c}\in\mathbb{R}^{d\times s}$ and $\mathbf{\Sigma_{c}}\in\mathbb{R}^{d\times d}$. The prior distribution for $\bm{{q}}_{u}$ is assumed to be standard normal:

[TABLE]

Let $\bm{{m}}_{0},\bm{{m}}_{u}\in\mathbb{R}^{kd}$, known as supervectors, denote vectors consisting of stacked global and utterance-specific component means $\bm{{\mu}}_{c}$ and $\bm{{\mu}}_{uc}$ respectively. Then, TVM can be summarized as:

[TABLE]

where $\mathbf{T}\in\mathbb{R}^{kd\times s}$ is given as: \mathbf{T}=\left[\BMAT(r){c.c.c}{c}\,\mathbf{T}_{1}^{\mathsf{T}}\,&\,\dots\,\,\mathbf{T}_{C}^{\mathsf{T}}\,\right]^{\mathsf{T}}

3 Algorithm

In this section, we describe an algorithm that uses TVM to adapt an NMF dictionary to the noise in an input spectrogram. The idea is for the dictionary to capture as much of the noise in the spectrogram as possible so that the activation matrix is not affected by noise. We will use the activation matrix as acoustic features for ASR on noisy speech. In our algorithm, the input features for the total variability model are magnitude spectrogram, the dictionary vectors play the role of GMM mean vectors, and the column-normalized activation matrix act as the posteriors of each GMM component. Table 1 summarizes how we fit NMF in the TVM paradigm, and the following subsections describe the algorithm in detail.

3.1 Step 1: Learning UBM Dictionary

We concatenate the utterances from the training set with various noisy conditions and compute the magnitude spectrogram $\mathbf{V}_{\text{all}}$. We use NMF to decompose $\mathbf{V}_{\text{all}}\in\mathbb{R}_{+}^{d\times t}$ into a $d\times k$ dictionary $\mathbf{W}_{\text{ubm}}$ and $k\times t$ activation matrix $\mathbf{H}_{\text{ubm}}$. We will refer to $\mathbf{W}_{\text{ubm}}$ as the UBM dictionary as it models the salient spectral components in the presence of all sources of variability. Thus, one can think of the $k$ vectors in the UBM dictionary as containing $k$ salient points in the feature space.

It has been reported in literature that incorporating a sparsity constraint on the activation matrix while applying NMF leads to a more expressive dictionary [18, 19]. Thus, we add an $\ell_{1}$ penalty on the activation matrix to the generalized KL divergence cost function to encourage sparsity:

[TABLE]

where $\hat{\mathbf{V}}_{\text{all}}\vcentcolon=\mathbf{W}_{\text{ubm}}{\mathbf{H}}_{\text{ubm}}$ and $\lambda$ controls the level of sparsity of $\mathbf{H}$. To minimize Equation 4, we iteratively update $\mathbf{W}_{\text{ubm}}$ and $\mathbf{H}_{\text{ubm}}$ with the following multiplicative updates:

[TABLE]

where $\odot$ means element-wise multiplication and the division is element-wise.

We stack the columns $\mathbf{W}_{\text{ubm}}$ to form the UBM dictionary supervector $\bm{{w}}_{\text{ubm}}$. This will act as the mean supervector for the rest of the steps.

3.2 Step 2: Calculation of Sufficient Statistics and Total Variability Matrix

In this step we calculate the 0th and 1st order sufficient statistics from each of the training files and use that to estimate the total variability matrix. We assume that the magnitude spectrograms $\mathbf{V}$ are drawn from a multivariate log-normal distribution, so $\log\left(\mathbf{V}\right)$ is drawn from a multivariate normal distribution. Thus, we will calculate the statistics for and model the total variability subspace of $\log\left(\mathbf{V}\right)$. Note that all the matrices involved have non-negative entries, so there are no issues when taking the log.

For each utterance $u$ in the training set, we calculate it’s magnitude spectrogram $\mathbf{V}_{u}\in\mathbb{R}_{+}^{d\times t_{u}}$. Using $\mathbf{W}_{\text{ubm}}$ as calculated in Step 1, we find the $k\times t_{u}$ activation matrix $\mathbf{H}_{u}$ as:

[TABLE]

where $\hat{\mathbf{V}}_{u}\vcentcolon=\mathbf{W}_{\text{ubm}}{\mathbf{H}}_{u}$. Define $\tilde{\mathbf{H}}_{u}$ as the columns of $\mathbf{H}_{u}$ normalized to sum to $1$. Then, each column of $\tilde{\mathbf{H}}_{u}$ represents the probability distribution of each time frame of $\mathbf{V}_{u}$ being represented by the vectors in the dictionary.

Next, we calculate the $0^{th}$-order statistic $\bm{{n}}_{u}$, $1^{st}$-order statistic $\mathbf{F}_{u}$, and centered $1^{st}$-order statistic $\bar{\mathbf{F}}_{u}$ for each training utterance $u$ as:

[TABLE]

where $\mathbf{N}_{u}\vcentcolon=\operatorname{diag}\left(\bm{{n}}_{u}\right)$ is a diagonal matrix formed by $\bm{{n}}_{u}$. Notice that $\bm{{n}}_{u}$ is a vector of size $k\times 1$ and $\mathbf{F}_{u}$ is a matrix of size $d\times k$, where $d$ is the dimension of the features.

We estimate the covariance matrix $\mathbf{\Sigma}_{c}$ as below:

[TABLE]

for $c\in\{1,\dots,k\}$. Once we have calculated the sufficient statistics for each input utterance, we use this to estimate the total variability matrix $\mathbf{T}$ by performing iterations of the EM algorithm. In each iteration, the posterior mean and covariance of the ivectors $\bm{{q}}_{u}$ are estimated during the E step given the current estimate of $\mathbf{T}$ as below:

[TABLE]

where $\bar{\bm{{f}}}_{u}$ is a supervector formed by stacking together columns of the matrix $\bar{\mathbf{F}}_{u}$ and $\mathbf{\Sigma}$ is the block-diagonal covariance matrix formed by $\mathbf{\Sigma}_{c}\;\forall c\in\left\{1,\dots,k\right\}$. Then, in the M step, the matrices $\mathbf{T}_{c}$ are updated as below:

[TABLE]

for $c\in\{1,\dots,k\}$.

3.3 Step 3: Extraction of Features

Given an utterance $a$, we find it’s magnitude spectrogram $\mathbf{V}_{a}$ and use this to calculate $\mathbf{H}_{a}$, $\tilde{\mathbf{H}}_{a}$, $\mathbf{n}_{a}$, and $\mathbf{F}_{a}$ using Equations 6 and 7. Then, given $\bm{{n}}_{a}$, $\mathbf{F}_{a}$, $\mathbf{T}$, $\bm{{w}}_{\text{ubm}}$, and $\mathbf{\Sigma}$, we obtain the i-vector $\bm{{q}}_{a}$ for this utterance using the posterior mean estimate given in Equation (9).

Thereafter, we calculate the adapted dictionary supervector (which now models the noise type and microphone factors because this information was captured by $\bm{{q}}_{a}$) as:

[TABLE]

Notice that $\bm{{w}}_{a}$ a $kd\times 1$ vector, so we reshape $\bm{{w}}_{a}$ to get a $d\times k$ non-negative adapted dictionary $\mathbf{W}_{a}$. With $\mathbf{W}_{a}$ fixed, we run the NMF algorithm on $\mathbf{V}_{a}$ to find the corresponding activation matrix $\mathbf{H}_{a}$. We use $\log\left(\mathbf{H}_{a}\right)$ as features for training the acoustic model in an ASR system.

4 Experiments and Results

We investigated the performance of our algorithm on the clean speech in the Aurora 4 corpus [20] with added noise from the DEMAND dataset [21]. The training set consists of 7138 utterances from the Aurora 4 training set corrupted by one of six different noises (labeled in the DEMAND dataset as “dliving”, “npark”, “omeeting”, “presto”, “straffic”, and “tcar”) at 5–15 dB SNR. The test set consists of 330 utterances from 8 speakers, with each of the utterances corrupted by the same six noises with SNRs ranging from 5–15 dB. Additionally, we created a second test set with “ohallway”, “pstation”, and “spsquare” noises added to test the ASR performance in unseen noise conditions.

We compared our proposed features to MFCCS, PNCCs [14], and the noise-robust features proposed in [15] (we will refer to these features as CNMF Features since that algorithm uses convolutive NMF (CNMF) to generate features). For our proposed features, we performed NMF on $d=129$-dimensional spectrograms with $k=60$ dictionary vectors. We set the dimension of the total variability subspace $s=400$. To match the parameters for the proposed features, we also used $60$ dictionary vectors for the CNMF Features. We extracted 13-dimensional MFCC and PNCC features. For each of the features, we applied a speaker-independent global mean and variance normalization prior to augmenting them with delta and delta-delta, followed by Linear Discriminant Analysis (LDA) and Maximum Linear Likelihood Transform (MLLT). We input the transformed features into a fully-connected 4-layer neural network, with 1024 hidden nodes per layer. The network uses tanh non-linearities and minimizes the cross-entropy using stochastic gradient descent.

Table 2 shows the word error rates (WER) on the Aurora 4 + DEMAND test set for our proposed features and the three baseline features for clean speech, noisy speech with noise seen during training, and noisy speech with noise not seen during training. We also provide the weighted average WER for the three conditions. From the results, one can see that clean speech has the lowest WER while the performance degrades with noise for all feature sets. At first glance, it seems surprising that the performance on unseen noises is better than on seen noises. But we note that in general the seen noises more background speech and non-stationary characteristics compared to the unseen noises, and these characteristics, particularly background speech, makes ASR more challenging. Indeed, when we inspected the results, we noticed that performance on “presto” noise (restaurant noise) had more than twice the WER compared to other noises.

Unfortunately we were not able achieve a lower WER with the proposed features compared to the baseline features. This is most likely due to the fact that the proposed features are computed from a per-utterance transform while MFCC, PNCC, and CNMF features are computed from fixed transforms, so the proposed features are sensitive to the parameters chosen when learning the transform and how well the total variability matrix $\mathbf{T}$ captures the sources of variability in the training set. In particular, the subspace dimension $s$ is very important because underestimating $s$ leads to poor modeling of the sources of variability, while overestimate $s$ will result in $\mathbf{T}$ capturing extraneous information. Moreover, we believe that $7138$ utterances in the training set may not be an adequate amount of observations to properly learn the total variability subspace. In fact, our initial experiments were carried out the standard Aurora 4 dataset, which includes microphone variability in addition to speaker and noise variability. Given the limited amount of training data, having additional variability due to channel conditions resulted WER that was much greater for Aurora 4 than for Aurora 4 + DEMAND111The WER for all feature sets was greater with Aurora 4 compared to Aurora 4 + DEMAND due to the multicondition style training. However, the amount of change in WER was much greater for the proposed features compared to the baseline features. Therefore, we are confident that a larger training set should result in a WER competitive with the other features as it will allow for better modeling of the total variability subspace.

On the other hand, one can see that the proposed features have the smallest gap in WER between the clean and unseen noise conditions compared to the baseline features. The main motivation behind this work is to generate acoustic features that are robust to acoustic conditions, so this result gives an indication that our algorithm can achieve this goal. This finding alone gives us good reason to improve the overall performance because it will allow ASR systems to perform near clean speech WER without requiring to re-train the acoustic model for specific acoustic conditions.

5 Conclusion

We proposed an algorithm to calculate noise-robust acoustic features from noisy utterances. The algorithm uses Total Variability Modeling to learn a total variability subspace and adapt a UBM NMF dictionary for each utterance at test time. We use the NMF activation matrix corresponding to the adapted dictionary as the acoustic features. Thus, our proposed features are calculated from per-utterance transforms, which could lead to greater robustness to the specific noise present in each utterance. Moreover, our algorithm, which builds upon the work in [15], does not require a training dataset of parallel clean and noisy speech. While the proposed features did not perform better than baseline features on the Aurora 4 + DEMAND corpus, we note that the WER was more consistent across clean and noisy conditions, in particular the unseen noise condition. This gives an indication that our approach has the potential to perform robustly in different noise conditions, but the overall results suggest that we should explore training with a larger dataset to better learn the total variability subspace.

Going forward, we will first retrain our system on a larger dataset, such as the Librispeech corpus [22]. Also, we will test the performance of the proposed algorithm in the presence of channel variability to more accurately simulate real-world acoustic conditions.