Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data

TL;DR

This paper explores pseudo-labeling techniques to improve joint speech transcription and translation in data-scarce scenarios, addressing domain mismatch issues with filtering and augmentation, leading to modest performance gains.

Contribution

It introduces domain mismatch remedies—pseudo-label filtering and data augmentation—for pseudo-labeling in joint speech transcription and translation, enhancing performance without extra supervision.

Findings

Up to 0.6% absolute WER improvement.

Up to 2.2 BLEU points increase.

Effective domain mismatch mitigation methods.

Abstract

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points.