Dual-View Predictive Diffusion: Lightweight Speech Enhancement via Spectrogram-Image Synergy

TL;DR

DVPD is a lightweight speech enhancement model that exploits spectrograms as both visual textures and frequency representations, achieving state-of-the-art results with significantly fewer parameters and computational costs.

Contribution

The paper introduces DVPD, a novel dual-view diffusion model that leverages spectral compression and visual feature extraction for efficient speech enhancement.

Findings

Achieves state-of-the-art performance on benchmarks.

Uses only 35% of parameters compared to SOTA models.

Reduces inference MACs by 60%.

Abstract

Diffusion models have recently set new benchmarks in Speech Enhancement (SE). However, most existing score-based models treat speech spectrograms merely as generic 2D images, applying uniform processing that ignores the intrinsic structural sparsity of audio, which results in inefficient spectral representation and prohibitive computational complexity. To bridge this gap, we propose DVPD, an extremely lightweight Dual-View Predictive Diffusion model, which uniquely exploits the dual nature of spectrograms as both visual textures and physical frequency-domain representations across both training and inference stages. Specifically, during training, we optimize spectral utilization via the Frequency-Adaptive Non-uniform Compression (FANC) encoder, which preserves critical low-frequency harmonics while pruning high-frequency redundancies. Simultaneously, we introduce a Lightweight Image-based Spectro-Awareness (LISA) module to capture features from a visual perspective with minimal overhead. During inference, we propose a Training-free Lossless Boost (TLB) strategy that leverages the same dual-view priors to refine generation quality without any additional fine-tuning. Extensive experiments across various benchmarks demonstrate that DVPD achieves state-of-the-art performance while requiring only 35% of the parameters and 40% of the inference MACs compared to SOTA lightweight model, PGUSE. These results highlight DVPD's superior ability to balance high-fidelity speech quality with extreme architectural efficiency. Code and audio samples are available at the anonymous website: {https://anonymous.4open.science/r/dvpd_demo-E630}