Enhancing time-frequency resolution with optimal transport and barycentric fusion of multiple spectrogram

TL;DR

This paper introduces a novel super-resolution spectrogram fusion method using optimal transport barycenters, allowing for improved time-frequency localization without requiring identical grids, validated on synthetic and speech signals.

Contribution

It proposes a new OT-based barycenter approach for spectrogram fusion that is flexible with input grids and computationally efficient, outperforming existing methods.

Findings

The method achieves sharper time-frequency localization.

It outperforms state-of-the-art unsupervised fusion techniques.

Validated on synthetic and speech signals with both quantitative and qualitative results.

Abstract

Time-frequency representations, such as the short-time Fourier transform (STFT), are fundamental tools for analyzing non-stationary signals. However, their ability to achieve sharp localization in both time and frequency is inherently limited by the Gabor-Heisenberg uncertainty principle. In this paper, we address this limitation by introducing a method to generate super-resolution spectrograms through the fusion of two or more spectrograms with varying resolutions. Specifically, we compute the super-resolution spectrogram as the barycenter of input spectrograms using optimal transport (OT) divergences. Unlike existing fusion approaches, our method does not require the input spectrograms to share the same time-frequency grid. Instead, the input spectrograms can be computed using any STFT parameters, and the resulting super-resolution spectrogram can be defined on an arbitrary user-specified grid. We explore various OT divergences based on different transportation costs. Notably, we introduce a novel transportation cost that preserves time-frequency geometry while significantly reducing computational complexity compared to standard Wasserstein barycenters. We adopt the unbalanced OT framework and derive a new block majorization-minimization algorithm for efficient barycenter computation. We validate the proposed method on controlled synthetic signals and recorded speech using both quantitative and qualitative evaluations. The results show that our approach combines the best localization properties of the input spectrograms and outperforms an unsupervised state-of-the-art fusion method.