Pseudo Label Is Better Than Human Label

TL;DR

This paper demonstrates that high-quality pseudo labels generated by a strong teacher model can outperform human labels in training automatic speech recognition systems, leading to significant improvements in word error rate.

Contribution

The authors develop a bi-directional teacher model using self-supervised and semi-supervised learning, achieving superior pseudo labels that enhance ASR performance beyond human-labeled data.

Findings

Teacher model achieves 4.0% WER on voice search.

Pseudo labels reduce WER by 13.6% relative for streaming models.

Pseudo labels outperform human labels in training effectiveness.

Abstract

State-of-the-art automatic speech recognition (ASR) systems are trained with tens of thousands of hours of labeled speech data. Human transcription is expensive and time consuming. Factors such as the quality and consistency of the transcription can greatly affect the performance of the ASR models trained with these data. In this paper, we show that we can train a strong teacher model to produce high quality pseudo labels by utilizing recent self-supervised and semi-supervised learning techniques. Specifically, we use JUST (Joint Unsupervised/Supervised Training) and iterative noisy student teacher training to train a 600 million parameter bi-directional teacher model. This model achieved 4.0% word error rate (WER) on a voice search task, 11.1% relatively better than a baseline. We further show that by using this strong teacher model to generate high-quality pseudo labels for training, we can achieve 13.6% relative WER reduction (5.9% to 5.1%) for a streaming model compared to using human labels.