Time-Frequency Trade-offs for Audio Source Separation with Binary Masks

TL;DR

This paper investigates how the choice of STFT window size affects binary mask-based audio source separation, revealing that optimal parameters depend on the specific types of signals being separated, such as speech or music.

Contribution

It demonstrates that the optimal STFT window size for binary mask separation varies with the source types, impacting separation quality in real-world audio signals.

Findings

Different window sizes are optimal for separating speech and music.

The trade-off between time and frequency resolution affects separation quality.

Implications for machine audition and learning applications.

Abstract

The short-time Fourier transform (STFT) provides the foundation of binary-mask based audio source separation approaches. In computing a spectrogram, the STFT window size parameterizes the trade-off between time and frequency resolution. However, it is not yet known how this parameter affects the operation of the binary mask in terms of separation quality for real-world signals such as speech or music. Here, we demonstrate that the trade-off between time and frequency in the STFT, used to perform ideal binary mask separation, depends upon the types of source that are to be separated. In particular, we demonstrate that different window sizes are optimal for separating different combinations of speech and musical signals. Our findings have broad implications for machine audition and machine learning in general.