Improving Resource-Efficient Speech Enhancement via Neural Differentiable DSP Vocoder Refinement

TL;DR

This paper introduces an efficient speech enhancement system for wearable devices that uses a neural network and a differentiable DSP vocoder to produce high-quality, real-time speech synthesis with minimal computational cost.

Contribution

It proposes a novel end-to-end framework combining neural acoustic feature prediction with a differentiable DSP vocoder for resource-efficient speech enhancement.

Findings

Improves speech intelligibility by 4% (STOI)

Enhances speech quality by 19% (DNSMOS)

Maintains low computational complexity suitable for real-time use

Abstract

Deploying speech enhancement (SE) systems in wearable devices, such as smart glasses, is challenging due to the limited computational resources on the device. Although deep learning methods have achieved high-quality results, their computational cost limits their feasibility on embedded platforms. This work presents an efficient end-to-end SE framework that leverages a Differentiable Digital Signal Processing (DDSP) vocoder for high-quality speech synthesis. First, a compact neural network predicts enhanced acoustic features from noisy speech: spectral envelope, fundamental frequency (F0), and periodicity. These features are fed into the DDSP vocoder to synthesize the enhanced waveform. The system is trained end-to-end with STFT and adversarial losses, enabling direct optimization at the feature and waveform levels. Experimental results show that our method improves intelligibility and quality by 4% (STOI) and 19% (DNSMOS) over strong baselines without significantly increasing computation, making it well-suited for real-time applications.