StutterZero and StutterFormer: End-to-End Speech Conversion for Stuttering Transcription and Correction

TL;DR

This paper introduces two novel end-to-end models, StutterZero and StutterFormer, that convert stuttered speech into fluent speech while transcribing, outperforming existing pipelines in accuracy and semantic similarity.

Contribution

The work presents the first end-to-end waveform-to-waveform models for simultaneous stutter correction and transcription, trained on synthesized paired data and evaluated on unseen speakers.

Findings

StutterZero reduces WER by 24% and improves BERTScore by 31%.

StutterFormer achieves a 28% decrease in WER and a 34% BERTScore improvement.

Both models outperform the leading Whisper-Medium model on benchmarks.

Abstract

Over 70 million people worldwide experience stuttering, yet most automatic speech systems misinterpret disfluent utterances or fail to transcribe them accurately. Existing methods for stutter correction rely on handcrafted feature extraction or multi-stage automatic speech recognition (ASR) and text-to-speech (TTS) pipelines, which separate transcription from audio reconstruction and often amplify distortions. This work introduces StutterZero and StutterFormer, the first end-to-end waveform-to-waveform models that directly convert stuttered speech into fluent speech while jointly predicting its transcription. StutterZero employs a convolutional-bidirectional LSTM encoder-decoder with attention, whereas StutterFormer integrates a dual-stream Transformer with shared acoustic-linguistic representations. Both architectures are trained on paired stuttered-fluent data synthesized from the SEP-28K and LibriStutter corpora and evaluated on unseen speakers from the FluencyBank dataset. Across all benchmarks, StutterZero had a 24% decrease in Word Error Rate (WER) and a 31% improvement in semantic similarity (BERTScore) compared to the leading Whisper-Medium model. StutterFormer achieved better results, with a 28% decrease in WER and a 34% improvement in BERTScore. The results validate the feasibility of direct end-to-end stutter-to-fluent speech conversion, offering new opportunities for inclusive human-computer interaction, speech therapy, and accessibility-oriented AI systems.