Fast MVAE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier

TL;DR

Fast MVAE (fMVAE) enhances multichannel source separation by significantly reducing computational time and improving classification accuracy through an auxiliary classifier VAE, maintaining competitive separation performance.

Contribution

The paper introduces fMVAE, a novel algorithm that combines an auxiliary classifier VAE with MVAE to improve efficiency and accuracy in source separation and classification.

Findings

Achieved 80% source classification accuracy.

Reduced computational time by approximately 93%.

Maintained comparable source separation performance to MVAE.

Abstract

This paper proposes an alternative algorithm for multichannel variational autoencoder (MVAE), a recently proposed multichannel source separation approach. While MVAE is notable in its impressive source separation performance, the convergence-guaranteed optimization algorithm and that it allows us to estimate source-class labels simultaneously with source separation, there are still two major drawbacks, i.e., the high computational complexity and unsatisfactory source classification accuracy. To overcome these drawbacks, the proposed method employs an auxiliary classifier VAE, an information-theoretic extension of the conditional VAE, for learning the generative model of the source spectrograms. Furthermore, with the trained auxiliary classifier, we introduce a novel algorithm for the optimization that is able to not only reduce the computational time but also improve the source classification performance. We call the proposed method "fast MVAE (fMVAE)". Experimental evaluations revealed that fMVAE achieved comparative source separation performance to MVAE and about 80% source classification accuracy rate while it reduced about 93% computational time.

Figures

Tables

Table 1. Table 1 : Average SDR, SIR and SAR scores of ILRMA, MVAE and fMVAE. The bold font shows the highest scores.

method	$R{T}_{60}$ = 78 ms
	SDR [dB]	SIR [dB]	SAR [dB]
ILRMA	14.8997	21.3277	18.0584
MVAE	21.5912	27.2663	25.1616
fMVAE	22.5976	29.8476	24.8967
method	$R{T}_{60}$ = 351 ms
	SDR [dB]	SIR [dB]	SAR [dB]
ILRMA	4.6840	11.6284	7.2364
MVAE	8.3157	18.0834	9.2206
fMVAE	6.7814	15.7728	7.7883

Table 2. Table 2 : Computational times of MVAE, fMVAE and ILRMA. MVAE and fMVAE were initialized with run ILRMA algorithm for 30 iterations in CPU and run 40 iterations of the optimization algorithms in CPU or GPU. ILRMA runs 100 iterations in CPU.

	rumtime/iteration[sec]	total [sec]
MVAE (GPU)	6.071632	260.5953
fMVAE (CPU)	0.389762	21.54129
fMVAE (GPU)	0.097272	17.56694
ILRMA (CPU)	0.113571	18.38221

Table 3. Table 3 : Accuracy rates of source classification obtained with MVAE and fMVAE.

	all iterations	final estimation
MVAE	27.91%	37.50%
fMVAE	78.63%	80.00%

Equations

$$\text{boldmathx}(f,n)$$

$$\text{boldmaths}(f,n)$$

$$\text{boldmathW}(f)$$

$$s_j(f,n)\sim {\mathcal{N}}_{\mathbb{C}}(s_j(f,n)|0,v_j(f,n)).$$

$$\text{boldmaths}(f,n)\sim {\mathcal{N}}_{\mathbb{C}}(\text{boldmaths}(f,n)|\text{boldmath0},\text{boldmathV}(f,n)),$$

$$\text{boldmathx}(f,n)\sim {\mathcal{N}}_{\mathbb{C}}(\text{boldmathx}(f,n)|\text{boldmath0},({\text{boldmathW}}^{\text{H}}(f){)}^{-1}\text{boldmathV}(f,n)\text{boldmathW}(f{)}^{-1}).$$

$$\log p(\mathcal{X}|$$

$$\displaystyle-\sum_{f,n,j}\Big{(}\log v_{j}(f,n)+\frac{|\textsf{\boldmath$w$}_{j}^{{\textsf{H}}}(f)\textsf{\boldmath$x$}(f,n)|^{2}}{v_{j}(f,n)}\Big{)},$$

$$\mathcal{J}(\varphi ,\theta )={\mathbb{E}}_{(\text{boldmathS},c)\sim p_D(\text{boldmathS},c)}$$

$$\text{leavevmode}\text{leavevmode}\text{leavevmode}-KL[q_{\varphi }(\text{boldmathz}|\text{boldmathS},c)||p(\text{boldmathz})]],$$

$$q_{\varphi }(\text{boldmathz}|\text{boldmathS},c)$$

$$p(\text{boldmathz})$$

$$p_{\theta }(\text{boldmathS}|\text{boldmathz},c,g)$$

$$v(f,n)$$

$$g_j\leftarrow \frac{1}{FN}\sum _{f,n}\frac{|y_j(f,n){|}^2}{{\sigma }_{\theta }^2(f,n;{\text{boldmathz}}_j,c_j)},$$

$$I(c,\text{boldmathS}|\text{boldmathz})$$

$$={\mathbb{E}}_{c\sim p(c),\text{boldmathS}\sim p_{\theta }(\text{boldmathS}|\text{boldmathz},c),c^{\prime }\sim p(c|\text{boldmathS})}[\log p(c^{\prime }|\text{boldmathS})]+H(c),$$

$${\mathbb{E}}_{c\sim p(c),\text{boldmathS}\sim p_{\theta }(\text{boldmathS}|\text{boldmathz},c),c^{\prime }\sim p(\text{boldmathS}|c)}[\log p(c^{\prime }|\text{boldmathS})]$$

$$=$$

$$\ge$$

$$=$$

$$\mathcal{L}(\varphi ,\theta ,\psi )$$

$$={\mathbb{E}}_{(\text{boldmathS},c)\sim p_D(\text{boldmathS},c),q_{\varphi }(\text{boldmathz}|\text{boldmathS},c)}[{\mathbb{E}}_{c\sim p(c),\text{boldmathS}\sim p_{\theta }(\text{boldmathS}|\text{boldmathz},c)}[\log r_{\psi }(c|\text{boldmathS})]].$$

$$\mathcal{I}(\psi )={\mathbb{E}}_{(\text{boldmathS},c)\sim p_D(\text{boldmathS},c)}[\log r_{\psi }(c|\text{boldmathS})]$$

$$\mathcal{J}(\varphi ,\theta )+{\lambda }_{\mathcal{L}}\mathcal{L}(\varphi ,\theta ,\psi )+{\lambda }_{\mathcal{I}}\mathcal{I}(\psi ),$$

Full text

**Index Terms— ** Multichannel source separation, multichannel variational autoencoder, auxiliary classifier, source classification

1 Introduction

Blind source separation (BSS) is a technique for separating out individual source signals from microphone array inputs when both the sources and the mixing methodology are unknown. The frequency-domain BSS approach allows us to perform instantaneous mixture separation and provides the flexibility of utilizing various models for the time-frequency representations of source signals. For example, independent vector analysis (IVA) [1, 2] solves frequency-wise source separation and permutation alignment simultaneously by assuming that the magnitudes of the frequency components originating from the same source tend to vary coherently over time. Multichannel extensions of non-negative matrix factorization (NMF), e.g., multichannel NMF (MNMF) [3, 4] and independent low-rank matrix analysis (ILRMA) [5, 6], provide an alternative solution to jointly solving these two problems by adopting NMF concept for the source spectrogram modeling. Specifically, the power spectrograms of the underlying source signals are approximated as the linear sum of a limited number of basis spectra scaled by time-varying amplitudes. It is noteworthy that IVA is equivalent to ILRMA in a particular case where only a single basis spectrum consisting of ones is used for each source signal. In this point of view, ILRMA can be interpreted as a generalized method of IVA that incorporates a source model with stronger representation power, which has shown to significantly improve the source separation performance [6].

Motived by this fact and the high capability of deep neural networks (DNNs) to spectrogram modeling, some attempts have recently been made to use DNNs for source models instead of the NMF model [7]–[12]. Multichannel variational autoencoder (MVAE) [11] is one of these methods that achieved great success in multi-speaker separation tasks. MVAE trains a conditional VAE (CVAE) [13, 14] using power spectrograms of clean speech samples and the corresponding speaker ID as auxiliary label inputs so that the trained decoder distribution can be used as a universal generative model of source signals, where the latent space variables and the class labels are the unknown parameters. At separation phase, MVAE iteratively updates the separation matrix using iterative projection (IP) method [15] and the unknown parameters of source generative model using backpropagation. The separated signals are obtained by applying the estimated separation matrix to the observed mixture signals. This optimization algorithm is notable in that the convergence to a stationary point is guaranteed and it allows estimating the source-class labels simultaneously with source separation. However, there are two major limitations. Firstly, the backpropagation needed for each iteration causes the optimization algorithm highly time-consuming, which can be troublesome in practical applications. Secondly, the encoder and decoder in a regular CVAE are free to ignore the class labels by finding networks that can reconstruct any data without using the additional information. In such a situation, the additional class labels will have limited effect on the spectrogram generation, which therefore leads to an unsatisfactory source classification result as we will show it in Section 4.

To address these limitations, this paper proposes “fast MVAE (fMVAE)” that employs an auxiliary classifier VAE (ACVAE) [16] for learning the generative distribution of source spectrograms and adopts the trained auxiliary classifier to the optimization at separation phase.

2 MVAE for determined multichannel source separation

2.1 Problem formulation

Let us consider a determined situation where $I$ source signals are captured by $I$ microphones. Let $x_{i}(f,n)$ and $s_{j}(f,n)$ denote the short-time Fourier transform (STFT) coefficients of the signal observed at the $i$-th microphone and the $j$-th source signal, where $f$ and $n$ are the frequency and time indices respectively. We denote the vectors containing $x_{1}(f,n),\ldots,x_{I}(f,n)$ and $s_{1}(f,n),\ldots,s_{I}(f,n)$ by

[TABLE]

where $(\cdot)^{{\textsf{T}}}$ denotes transpose. Under determined situation, the relationship between observed signals and source signals can be described as

[TABLE]

where \textsf{\boldmathW}^{{\textsf{H}}} is called the separation matrix. $(\cdot)^{{\textsf{H}}}$ denotes Hermitian transpose.

Let us assume that source signals follow the local Gaussian model (LGM), i.e., $s_{j}(f,n)$ independently follows a zero-mean complex Gaussian distribution with variance $v_{j}(f,n)={\mathbb{E}}[|s_{j}(f,n)|^{2}]$

[TABLE]

When $s_{j}(f,n)$ and s_{j^{\prime}}(f,n)$$(j\neq j^{\prime}) are independent, \textsf{\boldmaths}(f,n) follows

[TABLE]

where \textsf{\boldmathV}(f,n)=\mathop{\mathrm{diag}}\nolimits[v_{1}(f,n),\ldots,v_{I}(f,n)]. From (3) and (5), we can show that \textsf{\boldmathx}(f,n) follows

[TABLE]

Hence, the log-likelihood of the separation matrices \mathcal{W}=\{\textsf{\boldmathW}(f)\}_{f} and source model parameters $\mathcal{V}=\{v_{j}(f,n)\}_{j,f,n}$ given the observed mixture signals \mathcal{X}=\{\textsf{\boldmathx}(f,n)\}_{f,n} is given by

[TABLE]

where $\mathop{=}^{c}$ denotes equality up to constant terms. (2.1) will be split into frequency-wise source separation problems if there is no additional constraint imposed on $v_{j}(f,n)$. This indicates that there is a permutation ambiguity in the separated components for each frequency.

2.2 Multichannel VAE

To eliminate the permutation ambiguity during the estimation of $\mathcal{W}$, MVAE trains a conditional VAE (CVAE) to model the complex spectrograms \textsf{\boldmathS}=\{s(f,n)\}_{f,n} of source signals so that the spectral structures can be captured. CVAE consists of an encoder network q_{\phi}(\textsf{\boldmathz}|\textsf{\boldmathS},c) and a decoder network p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c), where the network parameters $\phi$ and $\theta$ are trained jointly using a set of labeled training samples \{\textsf{\boldmathS}_{m},c_{m}\}_{m=1}^{M}. Here, $c$ denotes the corresponding class label represented as a one-hot vector indicating to which class the spectrogram $S$ belongs. For example, if we consider speaker identities as the class category, each element of $c$ will be associated with a different speaker. CVAE is trained by maximizing the following variational lower bound

[TABLE]

where {\mathbb{E}}_{(\textsf{\boldmathS},c)\sim p_{D}(\textsf{\boldmathS},c)}[\cdot] denotes the sample mean over the training examples \{\textsf{\boldmathS}_{m},c_{m}\}_{m=1}^{M} and $KL[\cdot||\cdot]$ denotes Kullback–Leibler divergence. Here, the encoder distribution q_{\phi}(\textsf{\boldmathz}|\textsf{\boldmathS},c) and the prior distribution of the latent space variable p(\textsf{\boldmathz}) are expressed as Gaussian distributions

[TABLE]

where \textsf{\boldmath\mu}_{\phi}(\textsf{\boldmathS},c), \textsf{\boldmath\sigma}^{2}_{\phi}(\textsf{\boldmathS},c) are the encoder network outputs. The decoder distribution p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c) is defined as a zero-mean complex Gaussian distribution and a scale parameter $g$ is incorporated to eliminate the energy difference between the normalized training data and test data. Hence, the decoder distribution is expressed as

[TABLE]

where \sigma^{2}_{\theta}(f,n;\textsf{\boldmathz},c) denotes the $(f,n)$-th element of the decoder output. It is noteworthy to mention that the decoder distribution (12) is given in the same form as the LGM (5) so that the trained decoder distribution can be used as a universal generative model with ability to generate complex spectrograms belonging to all the source classes involved in the training examples. If we use p_{\theta}(\textsf{\boldmathS}_{j}|\textsf{\boldmathz}_{j},c_{j},g_{j}) to express the generative model of the complex spectrogram of the source $j$, a convergence-guaranteed optimization algorithm can be applied to search for a stationary point of the log-likelihood by iteratively updating (i) the separation matrices $\mathcal{W}$ using iterative projection (IP) method [15], (ii) the CVAE source model parameters \Psi=\{\textsf{\boldmathz}_{j},c_{j}\}_{j} using backpropagation, and (iii) the global scale parameter $\mathcal{G}=\{g_{j}\}_{j}$ with the following update rule

[TABLE]

where y_{j}(f,n)=\textsf{\boldmathw}_{j}^{\textsf{H}}(f)\textsf{\boldmathx}(f,n). Note that since the class labels are the model parameters estimated during the optimization, MVAE is able to perform source classification as well.

3 Proposed method: Fast MVAE

While MVAE is notable in that it works reasonably well for source separation and has capability to perform source classification simultaneously, there is still a huge room for improvement in the source classification performance. With a regular CVAE imposing no restrictions on the manner how the encoder and decoder may use the class labels, the encoder and decoder are free to ignore $c$ by finding distribution satisfying q_{\phi}(\textsf{\boldmathz}|\textsf{\boldmathS},c)=q_{\phi}(\textsf{\boldmathz}|\textsf{\boldmathS}) and p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c)=p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz}). As a result, $c$ will have little effect on generating source spectrograms that leads to a limited source classification performance. To avoid such situations, this paper proposes using an auxiliary classifier VAE [16] for learning the generative distribution p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c).

3.1 Auxiliary classifier VAE

Auxiliary classifier VAE (ACVAE) [16] is a variant of CVAE that incorporates an information-theoretic regularization [17] to assisting the decoder outputs to be correlated as far as possible with the class labels $c$ by maximizing the mutual information between $c$ and \textsf{\boldmathS}\sim p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c) conditioned on $z$. The mutual information is expressed as

[TABLE]

where $H(c)$ represents the entropy of $c$ that can be considered as a constant term. However, it is difficult to optimize I(c,\textsf{\boldmathS}|\textsf{\boldmathz}) directly since it requires access to the posterior p(c|\textsf{\boldmathS}). Fortunately, we can obtain a variational lower bound of the first term of I(c,\textsf{\boldmathS}|\textsf{\boldmathz}) by using a variational distribution r(c|\textsf{\boldmathS}) to approximate p(c|\textsf{\boldmathS}):

[TABLE]

the equality of which holds if r(c|\textsf{\boldmathS})=p(c|\textsf{\boldmathS}). We therefore can indirectly maximizing I(c,\textsf{\boldmathS}|\textsf{\boldmathz}) by increasing the lower bound with respect to p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c) and r(c|\textsf{\boldmathS}). One way to do this involves expressing the variational distribution as a neural network r_{\psi}(c|\textsf{\boldmathS}) and training it along with q_{\phi}(\textsf{\boldmathz}|\textsf{\boldmathS},c) and p_{\theta}(\textsf{\boldmathS}|\textsf{\boldmathz},c). r_{\psi}(c|\textsf{\boldmathS}) is called the auxiliary classifier. Therefore, the regularization term that we would like to maximize with respect to $\phi$, $\theta$, $\psi$ becomes

[TABLE]

Since the labeled training samples can also be used to train the auxiliary classifier r_{\psi}(c|\textsf{\boldmathS}), ACVAE also includes the cross-entropy

[TABLE]

in the training criterion. The entire training criterion is thus given by

[TABLE]

where $\lambda_{\mathcal{L}}\geq 0$ and $\lambda_{\mathcal{I}}\geq 0$ are weight parameters.

3.2 Fast algorithm

Note that the auxiliary classifier r_{\psi}(c|\textsf{\boldmathS}) not only assists the encoder and decoder to learn a more disentangled representation, but also provides an alternative to the backpropagation process in the original MVAE optimization, which is able to significantly reduce the computational time. We summarize the proposed fast algorithm as follows:

Train $\phi$, $\theta$ and $\psi$ using (19). 2. 2.

Initialize $\mathcal{W}$. 3. 3.

Iterate the following steps for each $j$:

1. (a)

Update $c_{j}$ using r_{\psi}(c_{j}|\textsf{\boldmathS}_{j}). 2. (b)

Update \textsf{\boldmathz}_{j} using q_{\phi}(\textsf{\boldmathz}_{j}|\textsf{\boldmathS}_{j},c_{j}). 3. (c)

Update $g_{j}$ using (14). 4. (d)

Update \textsf{\boldmathw}_{j}(0),\ldots,\textsf{\boldmathw}_{j}(F) using IP.

4 Experiments

To evaluate the effect of incorporating an auxiliary classifier into both the source model training and the optimization process, we conducted experiments to compare the multi-speaker separation performances, source classification accuracies and computational times of fMVAE and the conventional methods, i.e., ILRMA [5, 6] and MVAE [11].

4.1 Experimental conditions

We excerpted speech utterances from two male speakers (‘SM1’, ‘SM2’) and two female speakers (‘SF1’, ‘SF2’) from the Voice Conversion Challenge (VCC) 2018 dataset [18]. The audio files for each speaker were about 7 minutes long and manually segmented into 116 short sentences, where 81 and 35 sentences (about 5 and 2 minutes respectively) were used as training and test sets, respectively. The mixture signals were created by simulated two-channel recordings of two sources where the room impulse responses were synthesized using the image method. We tested two different reverberant conditions where the reverberation time ($RT_{60}$) was set at 78 ms and 351 ms, respectively. Fig. 1 shows the configuration of the room. We generated test data involving 4 speaker pairs and 10 sentences for each pair, namely there were totally 40 test signals, each of which was about 4 to 7 seconds long. All the speech signals were resampled at 16 kHz. The STFT was computed using Hamming window with 256 ms long and window shift was128 ms.

ILRMA was run for 100 iterations and both the proposed method and MVAE were run 40 iterations. To initialize $\mathcal{W}$ for MVAE and fMVAE, we used ILRMA run for 30 iterations. Adam [19] was used for training CVAE and ACVAE, and estimating the latent variables in MVAE. The network architectures used for CVAE and ACVAE is shown in Fig. 2. Note that we used the same network architectures of the encoder and decoder for CVAE and ACVAE. All the networks are designed as fully convolutional with gated linear units [20] so that the inputs are allowed to have arbitrary lengths. The programs were run using Intel (R) Core i7-6800K CPU@3.40 GHz and GeForce GTX 1080Ti GPU.

4.2 Results

We calculated the average of the signal-to-distortion ratios (SDR), signal-to-interference ratios (SIR) and signal-to-artifact ratios (SAR) [21] over the 40 test signals to evaluate the source separation performance and measured the computational times. Table 1 and Table 2 show the source separation results obtained under different $RT_{60}$ conditions and the computational times of ILRMA, MVAE and fMVAE, respectively. fMVAE was about 15 times faster than the original MVAE and even a little faster than ILRMA in GPU machines. Furthermore, it is noteworthy that fMVAE achieved comparative source separation performance with the original MVAE. As the results show, MVAE and fMVAE significantly outperformed ILRMA in terms of all the criteria of source separation performance, which confirmed the effect of the incorporation of the CVAE source model. fMVAE obtained higher SDR and SIR than MVAE under a low reverberant environment, but the performance slightly decreased when $RT_{60}$ became long. It is interesting to further compare fMVAE with MVAE in high reverberant environments, which is one direction of our future work.

To evaluation the performance of source classification, we computed the classification accuracy rates over the results estimated in each iteration and only in the final estimation. Table 3 shows the results that fMVAE significantly improved source classification accuracy with achieving 80% accuracy rate. Our future work also includes further improving the source classification accuracy.

5 Conclusions

This paper proposed fMVAE that (i) uses an auxiliary classifier VAE instead a regular CVAE for learning the generative distribution of source signals; (ii) employs the trained auxiliary classifier and encoder for the optimization. fMVAE allows us to significantly reduce the computational time and improve the source classification performance. The results revealed that fMVAE achieved about 15 times faster than the original MVAE and about 80% source classification accuracy rate with notable source separation performance.