WhispSynth: Scaling Multilingual Whisper Corpus through Real Data Curation and A Novel Pitch-free Generative Framework

TL;DR

WhispSynth is a large-scale, high-fidelity multilingual whispered speech corpus created through a novel generative framework combining DDSP-based pitch-free methods with TTS models, enabling improved whisper synthesis research.

Contribution

The paper introduces WhispSynth, a new high-quality whispered speech dataset generated with a novel pipeline that preserves vocal timbre and linguistic content, advancing multilingual whisper synthesis.

Findings

WhispSynth outperforms existing whispered speech corpora in quality.

CosyWhisper achieves naturalness comparable to real whispered speech.

The framework effectively combines DDSP and TTS for high-fidelity whisper data.

Abstract

Whisper generation is constrained by the difficulty of data collection. Because whispered speech has low acoustic amplitude, high-fidelity recording is challenging. In this paper, we introduce WhispSynth, a large-scale multilingual corpus constructed via a novel high-fidelity generative framework. Specifically, we propose a pipeline integrating Differentiable Digital Signal Processing (DDSP)-based pitch-free method with Text-to-Speech (TTS) models. This framework refines a comprehensive collection of resources, including our newly constructed WhispNJU dataset, into 118 hours of high-fidelity whispered speech from 479 speakers. Unlike standard synthetic or noisy real data, our data engine faithfully preserves source vocal timbre and linguistic content while ensuring acoustic consistency, providing a robust foundation for text-to-whisper research. Experimental results demonstrate that WhispSynth exhibits significantly higher quality than existing corpora. Moreover, our CosyWhisper, tuned with WhispSynth, achieves speech naturalness on par with ground-truth samples. The official implementation and related resources are available at https://github.com/tan90xx/cosywhisper.