Improving Deliberation by Text-Only and Semi-Supervised Training

TL;DR

This paper enhances an attention-based deliberation model by integrating text-only and semi-supervised training methods, leading to significant reductions in word error rate and improved human evaluation results.

Contribution

It introduces a novel approach combining text-only data and semi-supervised training into an attention-based deliberation model, improving speech recognition accuracy.

Findings

Achieved 4-12% WER reduction across various tasks.

Reduced Google Voice Search WER by 11% relative.

Positive human evaluation compared to state-of-the-art LM rescoring.

Abstract

Text-only and semi-supervised training based on audio-only data has gained popularity recently due to the wide availability of unlabeled text and speech data. In this work, we propose incorporating text-only and semi-supervised training into an attention-based deliberation model. By incorporating text-only data in training a bidirectional encoder representation from transformer (BERT) for the deliberation text encoder, and large-scale text-to-speech and audio-only utterances using joint acoustic and text decoder (JATD) and semi-supervised training, we achieved 4%-12% WER reduction for various tasks compared to the baseline deliberation. Compared to a state-of-the-art language model (LM) rescoring method, the deliberation model reduces the Google Voice Search WER by 11% relative. We show that the deliberation model also achieves a positive human side-by-side evaluation compared to the state-of-the-art LM rescorer with reasonable endpointer latencies.