Knowledge Distillation and Data Selection for Semi-Supervised Learning in CTC Acoustic Models

TL;DR

This paper introduces a semi-supervised learning approach for CTC-based speech recognition that emphasizes effective data selection to improve accuracy while reducing reliance on large unlabelled datasets.

Contribution

It proposes a novel data selection mechanism for semi-supervised CTC models, demonstrating significant WER improvements with less unlabelled data.

Findings

17% relative WER improvement over baseline

Effective data selection reduces need for large unlabelled datasets

Comparable performance with much larger random samples

Abstract

Semi-supervised learning (SSL) is an active area of research which aims to utilize unlabelled data in order to improve the accuracy of speech recognition systems. The current study proposes a methodology for integration of two key ideas: 1) SSL using connectionist temporal classification (CTC) objective and teacher-student based learning 2) Designing effective data-selection mechanisms for leveraging unlabelled data to boost performance of student models. Our aim is to establish the importance of good criteria in selecting samples from a large pool of unlabelled data based on attributes like confidence measure, speaker and content variability. The question we try to answer is: Is it possible to design a data selection mechanism which reduces dependence on a large set of randomly selected unlabelled samples without compromising on Word Error Rate (WER)? We perform empirical investigations of different data selection methods to answer this question and quantify the effect of different sampling strategies. On a semi-supervised ASR setting with 40000 hours of carefully selected unlabelled data, our CTC-SSL approach gives 17% relative WER improvement over a baseline CTC system trained with labelled data. It also achieves on-par performance with CTC-SSL system trained on order of magnitude larger unlabeled data based on random sampling.