Frequency-Weighted Training Losses for Phoneme-Level DNN-based Speech Enhancement

TL;DR

This paper introduces frequency-weighted SDR loss functions for DNN-based speech enhancement, improving perceptual quality and phoneme preservation by emphasizing critical spectral regions during training.

Contribution

It proposes novel perceptually-informed, frequency-dependent weighting schemes for SDR loss, enhancing phoneme-level speech intelligibility in multichannel enhancement models.

Findings

Frequency-weighted SDR losses improve perceptual speech quality.

Enhanced phoneme and consonant reconstruction observed.

Spectral and phoneme-level analysis confirms better cue preservation.

Abstract

Recent advances in deep learning have significantly improved multichannel speech enhancement algorithms, yet conventional training loss functions such as the scale-invariant signal-to-distortion ratio (SDR) may fail to preserve fine-grained spectral cues essential for phoneme intelligibility. In this work, we propose perceptually-informed variants of the SDR loss, formulated in the time-frequency domain and modulated by frequency-dependent weighting schemes. These weights are designed to emphasize time-frequency regions where speech is prominent or where the interfering noise is particularly strong. We investigate both fixed and adaptive strategies, including ANSI band-importance weights, spectral magnitude-based weighting, and dynamic weighting based on the relative amount of speech and noise. We train the FaSNet multichannel speech enhancement model using these various losses. Experimental results show that while standard metrics such as the SDR are only marginally improved, their perceptual frequency-weighted counterparts exhibit a more substantial improvement. Besides, spectral and phoneme-level analysis indicates better consonant reconstruction, which points to a better preservation of certain acoustic cues.