InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss

TL;DR

InterMPL introduces intermediate supervision into momentum pseudo-labeling for CTC-based speech recognition, significantly improving semi-supervised ASR performance by relaxing the independence assumption.

Contribution

The paper proposes a novel intermediate loss mechanism for MPL, enhancing CTC-based semi-supervised ASR by explicitly relaxing the independence assumption.

Findings

Up to 12.1% absolute performance gain in ASR accuracy.

Intermediate loss significantly improves MPL effectiveness.

Enhanced CTC models outperform traditional MPL in semi-supervised settings.

Abstract

This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision. Momentum PL (MPL) trains a connectionist temporal classification (CTC)-based model on unlabeled data by continuously generating pseudo-labels on the fly and improving their quality. In contrast to autoregressive formulations, such as the attention-based encoder-decoder and transducer, CTC is well suited for MPL, or PL-based semi-supervised ASR in general, owing to its simple/fast inference algorithm and robustness against generating collapsed labels. However, CTC generally yields inferior performance than the autoregressive models due to the conditional independence assumption, thereby limiting the performance of MPL. We propose to enhance MPL by introducing intermediate loss, inspired by the recent advances in CTC-based modeling. Specifically, we focus on self-conditional and hierarchical conditional CTC, that apply auxiliary CTC losses to intermediate layers such that the conditional independence assumption is explicitly relaxed. We also explore how pseudo-labels should be generated and used as supervision for intermediate losses. Experimental results in different semi-supervised settings demonstrate that the proposed approach outperforms MPL and improves an ASR model by up to a 12.1% absolute performance gain. In addition, our detailed analysis validates the importance of the intermediate loss.