WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation

TL;DR

WhispEar introduces a bidirectional framework that leverages pseudo-parallel whisper generation from normal speech to improve whispered-to-normal speech conversion, addressing data scarcity and enhancing performance.

Contribution

The paper presents a novel bidirectional model with zero-shot whisper generation, enabling scalable data augmentation for improved whisper-to-normal speech conversion.

Findings

Outperforms strong baselines in whisper-to-normal conversion

Scalable pseudo-parallel data improves model performance

Releases the largest bilingual whispered-normal corpus to date

Abstract

Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.