Multiple-hypothesis RNN-T Loss for Unsupervised Fine-tuning and Self-training of Neural Transducer

TL;DR

This paper introduces a multiple-hypothesis RNN-T loss function to improve unsupervised fine-tuning and self-training of speech recognition models, reducing errors caused by ASR hypothesis inaccuracies.

Contribution

It proposes a novel multiple-hypothesis loss for RNN-T models that mitigates ASR errors during unsupervised training and fine-tuning, outperforming single-hypothesis methods.

Findings

14.2% relative WER reduction on Librispeech test set

3.3% relative WER reduction on CHiME-4 noisy data

Effective in both fine-tuning and self-training scenarios

Abstract

This paper proposes a new approach to perform unsupervised fine-tuning and self-training using unlabeled speech data for recurrent neural network (RNN)-Transducer (RNN-T) end-to-end (E2E) automatic speech recognition (ASR) systems. Conventional systems perform fine-tuning/self-training using ASR hypothesis as the targets when using unlabeled audio data and are susceptible to the ASR performance of the base model. Here in order to alleviate the influence of ASR errors while using unlabeled data, we propose a multiple-hypothesis RNN-T loss that incorporates multiple ASR 1-best hypotheses into the loss function. For the fine-tuning task, ASR experiments on Librispeech show that the multiple-hypothesis approach achieves a relative reduction of 14.2% word error rate (WER) when compared to the single-hypothesis approach, on the test_other set. For the self-training task, ASR models are trained using supervised data from Wall Street Journal (WSJ), Aurora-4 along with CHiME-4 real noisy data as unlabeled data. The multiple-hypothesis approach yields a relative reduction of 3.3% WER on the CHiME-4's single-channel real noisy evaluation set when compared with the single-hypothesis approach.