Real-time Speech Restoration using Data Prediction Mean Flows

TL;DR

This paper introduces a real-time speech restoration model using data prediction mean flows, achieving high-quality results with minimal latency and significantly reduced computational requirements.

Contribution

The paper presents a novel low-latency flow matching architecture with data prediction mean flows that outperforms state-of-the-art models in efficiency while maintaining quality.

Findings

Uses 120x less compute than existing models.

Achieves similar audio quality without additional algorithmic latency.

Operates in real-time with low latency.

Abstract

Generative models are capable to address difficult problems with non-unique solutions like bandwidth extension and gap filling, removing highly non-linear artifacts from codecs, clipping and distortion, as opposed to removing linear additive components like noise and reverb. While large offline processing models have shown impressive results, these tasks have not been solved with real-time capable models with low latency and compute. We propose a few-step flow matching model using Data Prediction Mean Flows in combination with suitable novel low-latency architecture to make flow matching models an attractive choice under theses constraints. Compared to state-of-the-art, our proposed mean flow model uses 120x less compute and introduces no algorithmic latency other than the STFT, while achieving similar audio quality.