End-to-End Whisper to Natural Speech Conversion using Modified Transformer Network

TL;DR

This paper presents a novel end-to-end transformer-based method for converting whispered speech into natural speech, utilizing both parallel and non-parallel data, and introduces a new formant divergence metric for spectral shape comparison.

Contribution

It introduces an enhanced transformer architecture with auxiliary decoders for whisper-to-speech conversion, a new spectral shape measurement method, and demonstrates effectiveness on open datasets.

Findings

Reduced word error rate in ASR evaluations

Achieved high BLEU scores for speech quality

Formant distribution similarity to ground truth

Abstract

Machine recognition of an atypical speech like whispered speech, is a challenging task. We introduce whisper-to-natural-speech conversion using sequence-to-sequence approach by proposing enhanced transformer architecture, which uses both parallel and non-parallel data. We investigate different features like Mel frequency cepstral coefficients and smoothed spectral features. The proposed networks are trained end-to-end using supervised approach for feature-to-feature transformation. Further, we also investigate the effectiveness of embedded auxillary decoder used after N encoder sub-layers, trained with the frame-level objective function for identifying source phoneme labels. We show results on opensource wTIMIT and CHAINS datasets by measuring word error rate using end-to-end ASR and also BLEU scores for the generated speech. Alternatively, we also propose a novel method to measure spectral shape of it by measuring formant distributions w.r.t. reference speech, as formant divergence metric. We have found whisper-to-natural converted speech formants probability distribution is similar to the groundtruth distribution. To the authors' best knowledge, this is the first time enhanced transformer has been proposed, both with and without auxiliary decoder for whisper-to-natural-speech conversion and vice versa.