A unified convolutional beamformer for simultaneous denoising and dereverberation

TL;DR

This paper introduces a unified convolutional beamformer called WPD that optimally combines denoising and dereverberation, significantly enhancing speech quality and recognition accuracy.

Contribution

It presents a novel unified beamformer that integrates dereverberation and denoising into a single optimized framework, improving over traditional sequential methods.

Findings

Substantial improvement in speech enhancement metrics.

Enhanced automatic speech recognition performance.

Effective integration of dereverberation and denoising.

Abstract

This paper proposes a method for estimating a convolutional beamformer that can perform denoising and dereverberation simultaneously in an optimal way. The application of dereverberation based on a weighted prediction error (WPE) method followed by denoising based on a minimum variance distortionless response (MVDR) beamformer has conventionally been considered a promising approach, however, the optimality of this approach cannot be guaranteed. To realize the optimal integration of denoising and dereverberation, we present a method that unifies the WPE dereverberation method and a variant of the MVDR beamformer, namely a minimum power distortionless response (MPDR) beamformer, into a single convolutional beamformer, and we optimize it based on a single unified optimization criterion. The proposed beamformer is referred to as a Weighted Power minimization Distortionless response (WPD) beamformer. Experiments show that the proposed method substantially improves the speech enhancement performance in terms of both objective speech enhancement measures and automatic speech recognition (ASR) performance.

Tables

Table 1. TABLE I: Objective quality of enhanced speech evaluated using REVERB Challenge eval set. No Enh means no speech enhancement. Boldface indicates the best score for each metric.

	SimData				RealData
	CD	SRMR	FWSSNR	SIIB${}^{\text{Gauss}}$	SRMR
No Enh	3.97	3.68	3.62	241.2	3.18
WPE	3.76	4.77	4.99	315.3	5.00
MPDR	3.67	4.50	4.66	312.4	4.82
WPE+MPDR	3.01	5.37	7.52	486.8	6.57
Proposed	2.64	5.34	8.18	521.7	6.64

Table 2. TABLE II: Word error rate (WER) in % evaluated using REVERB Challenge eval set. No Enh means no speech enhancement. Boldface indicates the best score for each condition.

	SimData			RealData
	Near	Far	Average	Near	Far	Average
No Enh	4.18	6.25	5.22	17.53	19.68	18.61
WPE	4.04	4.90	4.47	12.33	13.88	13.11
MPDR	3.81	4.65	4.23	10.60	13.81	12.20
WPE+MPDR	4.00	4.69	4.35	8.75	11.31	10.03
Proposed	3.60	3.95	3.78	7.86	10.67	9.27

Equations

$${\mathbf{x}}_t=\sum _{{\tau }^{=}0}^{L_a}{\mathbf{a}}_{\tau }{s}_{t-\tau }+{\mathbf{n}}_t,$$

$${\mathbf{x}}_t$$

$${\mathbf{d}}_t$$

$${\mathbf{r}}_t$$

$${\mathbf{x}}_t=\sum _{\tau =b}^{L_w}{W}_{\tau }^{\mathsf{H}}{\mathbf{x}}_{t-\tau }+{\mathbf{d}}_t,$$

$$\hat{\bar{W}}=\underset{\bar{W}}{\mathrm{argmin}}\sum _t\frac{\Vert {\mathbf{x}}_t-{\sum }_{\tau =b}^{L_w}{W}_{\tau }^{\mathsf{H}}{\mathbf{x}}_{t-\tau }{\Vert }_2^2}{{\sigma }_t^2},$$

$${\hat{\mathbf{d}}}_t={\mathbf{x}}_t-\sum _{\tau =b}^{L_w}{\hat{W}}_{\tau }^{\mathsf{H}}{\mathbf{x}}_{t-\tau }.$$

$${\mathbf{x}}_t=\mathbf{v}{s}_t+{\mathbf{n}}_t.$$

$${\hat{\mathbf{w}}}_0=\underset{{\mathbf{w}}_0}{\mathrm{argmin}}\sum _t{\left|{\mathbf{w}}_0^{\mathsf{H}}{\mathbf{x}}_t\right|}^{2}s.t.{\mathbf{w}}_0^{\mathsf{H}}\mathbf{v}=1.$$

$${\hat{d}}_t^{(q)}={\hat{\mathbf{w}}}_0^{\mathsf{H}}{\mathbf{x}}_t.$$

$${\hat{d}}_t^{(q)}$$

$$={\mathbf{w}}_0^{\mathsf{H}}{\mathbf{x}}_t+\sum _{\tau =b}^{L_w}{\mathbf{w}}_{\tau }^{\mathsf{H}}{\mathbf{x}}_{t-\tau },$$

$$={\bar{\mathbf{w}}}^{\mathsf{H}}{\bar{\mathbf{x}}}_t,$$

$$\widehat{\bar{\mathbf{w}}}=\underset{\bar{\mathbf{w}}}{\mathrm{argmin}}\sum _t\frac{|{\bar{\mathbf{w}}}^{\mathsf{H}}{\bar{\mathbf{x}}}_t{|}^2}{{\sigma }_t^2}s.t.{\mathbf{w}}_0^{\mathsf{H}}\mathbf{v}=1.$$

$$\widehat{\bar{\mathbf{w}}}=\frac{R^{-1}\bar{\mathbf{v}}}{{\bar{\mathbf{v}}}^{\mathsf{H}}{R}^{-1}\bar{\mathbf{v}}},$$

$$R=\sum _t\frac{{\bar{\mathbf{x}}}_t{\bar{\mathbf{x}}}_t^{\mathsf{H}}}{{\sigma }_t^2}.$$

$${\hat{d}}_t^{(q)}={\widehat{\bar{\mathbf{w}}}}^{\mathsf{H}}{\bar{\mathbf{x}}}_t.$$

$${\hat{d}}_t^{(q)}={\mathbf{w}}_0^{\mathsf{H}}{\mathbf{x}}_t-\sum _{\tau =b}^{L_w}{\mathbf{c}}_{\tau }^{\mathsf{H}}(W_0^{\mathsf{H}}{\mathbf{x}}_{t-\tau }),$$

Full text

I Introduction

When a speech signal is captured by distant microphones, e.g., in a conference room, it will inevitably contain additive noise and reverberation components. These components are detrimental to the perceived quality of the observed speech signal and often cause serious degradation in many applications such as hands-free teleconferencing and automatic speech recognition (ASR).

Microphone array signal processing techniques have been developed to minimize the aforementioned detrimental effects by reducing the noise and the reverberation in the acquired signal. A filter-and-sum beamformer [1], a minimum-variance distortionless response (MVDR) beamformer and a minimum-power distortionless response (MPDR) beamformer [2, 3, 4, 5, 6], and a maximum signal-to-noise ratio beamformer [7, 8, 9] are widely-used systems for denoising, while a weighted prediction error (WPE) method and its variants [10, 11, 12, 13, 14] are emerging techniques for dereverberation. The usefulness of these techniques, particularly for improving ASR performance, has been extensively studied, e.g., at the REVERB challenge [15] and the CHiME-3/4/5 challenges [16, 17, 18]. Advances in this technological area have led to recent progress on commercial devices with far-field ASR capability, such as smart speakers [19, 20, 21].

However, it remains a challenge to reduce both noise and reverberation simultaneously in an optimal way. For example, researchers have proposed using MVDR beamforming and WPE dereverberation in a cascade manner [22, 23], where, for example, the signal is first processed by WPE dereverberation and then denoised with MVDR beamforming. With this approach, dereverberation may not be optimal due to the influence of the noise, and denoising may be disturbed by the remaining reverberation. Certain joint optimization techniques have also been proposed [24, 25, 26], but they perform dereverberation and denoising separately, which makes the optimality of the integration unclear, resulting in marginal performance improvement compared with the cascade system.

To achieve optimal integration, this paper proposes a method for unifying WPE dereverberation and MPDR beamforming, into a single convolutional beamforming approach and for optimizing the beamformer based on a single unified optimization criterion. We can derive a closed-form solution for this beamformer, assuming that the time-varying power and steering vector of the desired signal are given. The optimality of the beamformer is guaranteed under the assumed optimization criterion and condition. The beamformer is referred to as a Weighted Power minimization Distortionless response (WPD) beamformer. Note that the steering vector and the signal power must also be given for WPE dereverberation and MPDR beamforming, respectively, and several techniques for their estimation have already been proposed [25, 27, 28].

In the experiments, we compare the proposed method with WPE dereverberation, MPDR beamforming, and both approaches in a cascade configuration in terms of objective speech enhancement measures and ASR performance. The experiments show that the proposed method substantially outperforms all the conventional methods with regard to almost all the performance metrics. For example, in comparison with the cascade system, the proposed method achieves an average word error reduction rate of 7.5 % for real data taken from the REVERB Challenge dataset.

II Signal model

Assume that a single speech signal is captured by $M$ microphones in a noisy reverberant environment. Then, the captured signal in the short time Fourier transform (STFT) domain is approximately modeled at each frequency bin by

[TABLE]

where $t$ and $\tau$ are time frame indices. Note that all the symbols should also have frequency bin indices, but they are omitted for brevity in this paper assuming that each frequency bin is processed independently in the same way. Letting $\top$ denote the non-conjugate transpose, $\mathbf{x}_{t}=[x_{t}^{(1)},x_{t}^{(2)},\ldots,x_{t}^{(M)}]^{\top}$ is a column vector containing the STFT coefficients of the captured signals for all the microphones at a time frame $t$, $s_{t}$ is an STFT coefficient of clean speech signal at a time frame $t$, $\mathbf{a}_{t}=[a_{t}^{(1)},a_{t}^{(2)},\ldots,a_{t}^{(M)}]^{\top}$ for $t=0,1,\ldots,L$ is a sequence of column vectors containing convolutional acoustic transfer functions (ATFs) from the speaker location to all the microphones, $L_{a}$ is the length of the convolutional ATFs in each frequency bin, and $\mathbf{n}_{t}=[n_{t}^{(1)},n_{t}^{(2)},\ldots,n_{t}^{(M)}]^{\top}$ is the additive noise. As in eq. (1), according to [29], the effect of the reverberation can be approximately represented by the convolution in the STFT domain between $s_{t}$ and $\mathbf{a}_{t}$ when the length of the room impulse response in the time domain is longer than the analysis window. Hereafter, we refer to a sequence of STFT coefficients in each frequency bin, such as $x_{t}^{(m)}$ and $s_{t}$ for $t=1,2,\ldots$, simply as a signal.

The first term in eq. (1) can be further decomposed into two parts, one composed of a direct signal and early reflections, hereafter referred to as the desired signal $\mathbf{d}_{t}$, and the other corresponding to the late reverberation $\mathbf{r}_{t}$ [30]. With this decomposition, eq. (1) is rewritten as

[TABLE]

where $b$ is the frame index that divides the convolutional ATFs into the ATF coefficients for $\mathbf{d}_{t}$ and those for $\mathbf{r}_{t}$. Later, $b$ is also termed the prediction delay for WPE dereverberation and WPD beamforming. Finally, we define the goal of realizing speech enhancement to preserve $\mathbf{d}_{t}$ while reducing $\mathbf{r}_{t}$ and $\mathbf{n}_{t}$ from $\mathbf{x}_{t}$.

III Conventional methods

This section gives a brief overview of the conventional methods, including WPE dereverberation, MPDR beamforming, and two approaches with a cascade configuration.

III-A Dereverberation by WPE

If we disregard the additive noise, $\mathbf{n}_{t}$, we can rewrite eq. (1) using a multichannel autoregressive model [31, 32, 10] as

[TABLE]

where $L_{w}$ is the regression order, $\mathsf{H}$ denotes the conjugate transpose, $W_{t}$ for $t=b,b+1,\ldots,{L_{w}}$ are $M\times M$ dimensional matrices containing coefficients that predict the current captured signal, $\mathbf{x}_{t}$, from the past captured signals, $\mathbf{x}_{t-\tau}$ for $\tau=b,b+1,\ldots,{L_{w}}$, and the second term in the equation, referred to as the prediction error, is assumed to be the desired signal according to the model [10].

WPE dereverberation estimates the prediction coefficients based on maximum likelihood estimation, assuming that the desired signal at each microphone follows a time-varying complex Gaussian distribution with a mean of zero and a time-varying variance, $\sigma_{t}^{2}$, which corresponds to the time-varying power of the desired signal. Then, the prediction coefficients, $\bar{W}=[W_{b},W_{b+1},\ldots,W_{L_{w}}]^{\top}$, are estimated as those that minimize the average power of the prediction error weighted by the inverse of $\sigma_{t}^{2}$. The estimation is represented by

[TABLE]

where $||\mathbf{x}||_{2}^{2}=\mathbf{x}^{\mathsf{H}}\mathbf{x}$ is the squared $L_{2}$ norm of a vector $\mathbf{x}$. It is known that the prediction delay $b$ also works as a distortionless constraint to prevent the desired signal components from being distorted by the dereverberation [10]. As for the estimation of $\sigma_{t}^{2}$, several useful techniques have been proposed including an iterative estimation method [29, 13].

With the estimated prediction coefficients, the dereverberation is performed by

[TABLE]

It was experimentally confirmed that WPE dereverberation can function robustly even in noisy environments to reduce the late reverberation with a slight increase in the noise [10].

III-B Beamforming by MPDR

Assuming that the desired signal can be approximated as the product of a vector $\mathbf{v}$ with a clean speech signal, i.e., $\mathbf{d}_{t}=\mathbf{v}s_{t}$, and taking the late reverberation, $\mathbf{r}_{t}$, as part of the noise, $\mathbf{n}_{t}$, eq. (2) becomes

[TABLE]

The MPDR beamformer is defined as a vector, $\mathbf{w}_{0}$, that minimizes the average power of the captured signal, $\mathbf{x}_{t}$, under a distortionless constraint, $\mathbf{w}_{0}^{\mathsf{H}}\mathbf{v}=1$, that keeps the clean speech, $s_{t}$, unchanged by the beamforming [2, 3]. Here, $\mathbf{v}$ is also termed a steering vector, and techniques for its estimation from a captured signal have been proposed. Due to the scale ambiguity in the steering vector estimation, in practice it is substituted by a relative transfer function (RTF) [33]. An RTF is defined as the steering vector normalized by its value at a reference channel, calculated by $\mathbf{v}/v^{(q)}$ where $v^{(q)}$ denotes the value at the reference channel. This makes the distortionless constraint work to keep the desired signal at the reference channel, $d_{t}^{(q)}$, unchanged.

The beamformer is estimated as follows:

[TABLE]

The desired signal is then estimated as

[TABLE]

With the beamformer, the resultant signal is composed of only one channel signal corresponding to the reference channel $q$.

On the basis of the above discussion, MPDR beamforming can perform both denoising and dereverberation [34] by reducing $\mathbf{n}_{t}$, which contains the additive noise and the late reverberation. However, its dereverberation capability is limited because it cannot reduce reverberation components that come from the target speaker direction, especially when there are few microphones.

III-C Cascade of WPE dereverberation and MPDR beamforming

To achieve better speech enhancement in noisy reverberant environments, researchers have proposed using both WPE dereverberation and MPDR beamforming in a cascade configuration [22]. Because WPE dereverberation can dereverberate all the microphone signals individually, MPDR beamforming can be applied after WPE dereverberation has been applied. Techniques have also been proposed for estimating the steering vector and the power of the desired signal, for example, by iteratively and alternately applying WPE dereverberation and MPDR beamforming to the signals [25].

IV Proposed method

This section describes a method for unifying WPE dereverberation and MPDR beamforming into a single convolutional beamforming approach. A closed-form solution can be obtained for the beamformer given the steering vector and the time-varying power of the desired signal, and we can perform more effective speech enhancement than with a simple cascade consisting of WPE dereverberation and MPDR beamforming. Figure 1 illustrates the processing flow of the method.

IV-A Convolutional beamforming by WPD

First, the signal obtained using the cascade consisting of WPE dereverberation and MPDR beamforming, i.e., eqs. (7) and (10), can be rewritten as

[TABLE]

where we set $\mathbf{w}_{t}=-W_{t}\mathbf{w}_{0}$ to obtain the second line above, and we set $\bar{\mathbf{w}}=[\mathbf{w}_{0}^{\top},\mathbf{w}_{b}^{\top},\mathbf{w}_{b+1}^{\top},\ldots,\mathbf{w}_{L_{w}}^{\top}]^{\top}$ and $\bar{\mathbf{x}}_{t}=[\mathbf{x}_{t}^{\top},\mathbf{x}_{t-b}^{\top},\mathbf{x}_{t-b-1}^{\top},\ldots,\mathbf{x}_{t-L_{w}+1}^{\top}]^{\top}$ to obtain the third line. Note that $\bar{\mathbf{w}}$ and $\bar{\mathbf{x}}_{t}$ contain a time gap between their first and the second elements, corresponding to the prediction delay $b$.

Next, the optimization criterion is defined based on the model of the desired speech used for WPE dereverberation, namely the time-varying Gaussian distribution, and based on the distortionless constraint used for MPDR beamforming. Specifically, we estimate the convolutional filter, $\bar{\mathbf{w}}$, as one that minimizes the average weighted power of a signal under a distortionless constraint. It is represented by

[TABLE]

Here, all the filter coefficients are optimized based on the average weighted power minimization criterion. Note that the use of the time-varying weight makes the distribution of the enhanced speech obtained by beamforming closer to that of the desired speech.

Eq. (14) can be viewed as a variation of eq. (9), which is used for conventional MPDR beamforming. Unlike eq. (9), eq. (14) evaluates the average weighted power of the signal, and considers both the spatial and temporal covariance. The solution is obtained as follows:

[TABLE]

where $\bar{\mathbf{v}}=[\mathbf{v}^{\top},0,0,\ldots,0]^{\top}$ is a column vector containing $\mathbf{v}$ followed by $M(L_{w}-b+1)$ zeros, and $R$ is a power-normalized temporal-spatial covariance matrix with a prediction delay, which is defined as

[TABLE]

Finally, with the estimated convolutional filter, $\hat{\bar{\mathbf{w}}}$, the target speech is estimated as

[TABLE]

Interestingly, the same solution can be derived for the proposed method even when we concatenate MPDR beamforming and WPE dereverberation in reverse order. The signal obtained in this case becomes

[TABLE]

where $\mathbf{w}_{0}$ is the MPDR beamformer applied to $\mathbf{x}_{t}$, $W_{0}$ is an arbitrary denoising matrix that contains $\mathbf{w}_{0}$ in its first column, and $\mathbf{c}_{t}$ is a coefficient vector that predicts the current denoised signal, $\mathbf{w}_{0}^{\mathsf{H}}\mathbf{x}_{t}$, from the past denoised signals, $W_{0}^{\mathsf{H}}\mathbf{x}_{t-\tau}$. Then, eq. (12) is obtained by setting $\mathbf{w}_{t}=-W_{0}\mathbf{c}_{t}$, and optimized in the way discussed above.

V Experiments

V-A Dataset and evaluation metrics

We evaluated the performance of the proposed method using the REVERB Challenge dataset [15]. The evaluation set (Eval set) of the dataset is composed of simulated data (SimData) and real recordings (RealData). Each utterance in the dataset contains reverberant speech uttered by a speaker and stationary additive noise. The distance between the speaker and the microphone array is varied from 0.5 m to 2.5 m. For SimData, the reverberation time is varied from about 0.25 s to 0.7 s, and the signal-to-noise ratio (SNR) is set at about 20 dB.

As objective measures for evaluating speech enhancement performance [35], we used the cepstrum distance (CD), the frequency-weighted segmental SNR (FWSSNR), the speech-to-reverberation modulation energy ratio (SRMR) [36], and the speech intelligibility in bits with the information capacity of a Gaussian channel (SIIB${}^{\mbox{\tiny Gauss}}$) [37]. SIIB${}^{\mbox{\tiny Gauss}}$ is a recently proposed intrusive instrumental metric that is used to evaluate the intelligibility of distorted speech signals. To evaluate the enhanced speech in terms of ASR performance, we used a baseline ASR system recently developed using kaldi [38]. This is a fairly competitive system composed of a time-delay neural network acoustic model trained using a lattice-free maximum mutual information criterion and online i-vector extraction, and a tri-gram language model.

V-B Methods to be compared and analysis conditions

We compared WPD beamforming (Proposed) with WPE dereverberation, MPDR beamforming, and WPE dereverberation followed by MPDR beamforming (WPE+MPDR). For all the methods, a hanning window was used for a short time analysis with the frame length and the shift set at 32 ms and 8 ms, respectively. The sampling frequency was 16 kHz and $M=8$ microphones were used. For WPE dereverberation, WPE+MPDR, and WPD beamforming, the prediction delay was set at $b=4$, and the order of the autoregressive model was set at $L_{w}=12,10,8$, and $6$, respectively, for frequency ranges of [math] to $0.8$ kHz, $0.8$ to $1.5$ kHz, $1.5$ to $3$ kHz, $3$ to $6$ kHz, and $6$ to $8$ kHz.

The time-varying power, $\sigma_{t}^{2}$, and the steering vector, $\mathbf{v}$ were estimated from the captured signal based on a method used in [25]. Figure 2 shows the processing flow. The same estimates were used for all the methods. Adopting the power of the captured signal as the initial value of $\sigma_{t}^{2}$, we repeatedly applied WPE+MPDR to the captured signal, and updated $\mathbf{v}$ and $\sigma_{t}^{2}$ using the outputs of the WPE dereverberation and MPDR beamforming, respectively. The number of iterations was set at two. The steering vector was estimated based on the generalized eigenvalue decomposition with covariance whitening [27, 28] assuming that each utterance has noise-only periods of 225 ms and 75 ms, respectively, at its beginning and ending parts.

V-C Evaluation with objective speech enhancement measures

Table I summarizes evaluation results obtained using objective speech enhancement measures. First, all the methods improved the speech quality with all the measures. In addition, WPE+MPDR greatly outperformed WPE dereverberation and MPDR beamforming, while the proposed method further outperformed WPE+MPDR for all the metrics except for SRMR on SimData. These results clearly show the superiority of WPD beamforming.

V-D Evaluation using ASR

Table II shows the word error rates (WERs) obtained using the baseline ASR system. The proposed method greatly outperformed all the other methods under all the conditions.

Finally, it may be interesting to compare WPD beamforming roughly111The analysis conditions used for the two methods, such as the length of the convolutional filter and the way of calculating $\sigma_{t}^{2}$ and $\mathbf{v}$, are not the same. with the frontend of the best performing system [22] at the REVERB challenge. The frontend was composed of WPE dereverberation and MVDR beamforming followed by a nonlinear denoising method, DOLPHIN [39]. With this frontend and the kaldi ASR baseline, the average WERs for RealData were 10.29 and 9.07 % w/o and w/ DOLPHIN, respectively. In contrast, when we evaluated WPD beamforming w/o and w/ DOLPHIN, the WERs were 9.27 and 8.91 %, respectively. This again indicates the superiority of WPD beamforming.

VI Concluding remarks

This paper presented a method for unifying WPE dereverberation and MPDR beamforming that made it possible to perform denoising and dereverberation both optimally and simultaneously based on microphone array signal processing. Convolutional beamforming by WPD was derived and shown to improve the speech enhancement performance in noisy reverberant environments, with regard to objective speech enhancement measures and WERs, in comparison with conventional methods, including WPE dereverberation, MPDR beamforming, and WPE+MPDR. Future work will include an evaluation of WPD beamforming in various environments, the introduction of different optimization criteria, and the extension of the proposed method to online processing.