Leveraging Heteroscedastic Uncertainty in Learning Complex Spectral Mapping for Single-channel Speech Enhancement

TL;DR

This paper introduces a novel speech enhancement method that models heteroscedastic uncertainty using a multivariate Gaussian likelihood, leading to improved performance by adaptively weighting the loss based on uncertainty estimates.

Contribution

The paper proposes a new approach that incorporates heteroscedastic uncertainty modeling into spectral mapping for speech enhancement, outperforming traditional loss functions.

Findings

Modeling heteroscedastic uncertainty improves SE performance.

The approach outperforms MSE, MAE, and SI-SDR loss functions.

Weakening covariance assumptions enhances the effectiveness of the NLL loss.

Abstract

Most speech enhancement (SE) models learn a point estimate and do not make use of uncertainty estimation in the learning process. In this paper, we show that modeling heteroscedastic uncertainty by minimizing a multivariate Gaussian negative log-likelihood (NLL) improves SE performance at no extra cost. During training, our approach augments a model learning complex spectral mapping with a temporary submodel to predict the covariance of the enhancement error at each time-frequency bin. Due to unrestricted heteroscedastic uncertainty, the covariance introduces an undersampling effect, detrimental to SE performance. To mitigate undersampling, our approach inflates the uncertainty lower bound and weights each loss component with their uncertainty, effectively compensating severely undersampled components with more penalties. Our multivariate setting reveals common covariance assumptions such as scalar and diagonal matrices. By weakening these assumptions, we show that the NLL achieves superior performance compared to popular loss functions including the mean squared error (MSE), mean absolute error (MAE), and scale-invariant signal-to-distortion ratio (SI-SDR).