Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition

TL;DR

This paper introduces a task-agnostic training approach using domain adversarial learning to improve speech recognition accuracy for both monolingual and code-switched speech, addressing data scarcity and performance deterioration issues.

Contribution

It proposes a novel domain adversarial training method that creates shared representations for monolingual and code-switched speech recognition, enhancing performance across multiple language pairs.

Findings

Reductions in Word Error Rates for monolingual and code-switched speech

Shared layer parameters learned by adversarial discriminator are task-agnostic

Improved robustness of ASR systems across different language scenarios

Abstract

Recognizing code-switched speech is challenging for Automatic Speech Recognition (ASR) for a variety of reasons, including the lack of code-switched training data. Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize both monolingual and code-switched speech with high accuracy. Our experiments indicated that this loss in performance could be mitigated by using certain strategies for fine-tuning and regularization, leading to improvements in both monolingual and code-switched ASR. In this work, we present further improvements over our previous work by using domain adversarial learning to train task agnostic models. We evaluate the classification accuracy of an adversarial discriminator and show that it can learn shared layer parameters that are task agnostic. We train end-to-end ASR systems starting with a pooled model that uses monolingual and code-switched data along with the adversarial discriminator. Our proposed technique leads to reductions in Word Error Rates (WER) in monolingual and code-switched test sets across three language pairs.