Optimizing Alignment of Speech and Language Latent Spaces for End-to-End Speech Recognition and Understanding

TL;DR

This paper introduces a novel method to align speech and text latent spaces in end-to-end speech recognition systems, significantly improving WER and SLU performance by using embedding alignment and modality switch training.

Contribution

It proposes an embedding aligner and modality switch training to better unify speech and text representations in E2E ASR systems, addressing the mismatch problem.

Findings

Achieves 14-19% relative WER reduction on Librispeech.

Improves SLU slot filling F1 score by 2.5-2.8%.

Demonstrates effective alignment of speech and text latent spaces.

Abstract

The advances in attention-based encoder-decoder (AED) networks have brought great progress to end-to-end (E2E) automatic speech recognition (ASR). One way to further improve the performance of AED-based E2E ASR is to introduce an extra text encoder for leveraging extensive text data and thus capture more context-aware linguistic information. However, this approach brings a mismatch problem between the speech encoder and the text encoder due to the different units used for modeling. In this paper, we propose an embedding aligner and modality switch training to better align the speech and text latent spaces. The embedding aligner is a shared linear projection between text encoder and speech encoder trained by masked language modeling (MLM) loss and connectionist temporal classification (CTC), respectively. The modality switch training randomly swaps speech and text embeddings based on the forced alignment result to learn a joint representation space. Experimental results show that our proposed approach achieves a relative 14% to 19% word error rate (WER) reduction on Librispeech ASR task. We further verify its effectiveness on spoken language understanding (SLU), i.e., an absolute 2.5% to 2.8% F1 score improvement on SNIPS slot filling task.