Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR

TL;DR

This paper introduces two novel methods for explicitly estimating the internal language model in attention-based encoder-decoder ASR systems, improving external language model integration and outperforming previous approaches.

Contribution

The paper proposes two new techniques for ILM estimation in LAS-based ASR, enhancing accuracy and external LM fusion effectiveness.

Findings

Achieved lowest perplexity for ILMs among tested methods.

Significantly outperformed shallow fusion and previous ILME approaches.

Validated on multiple datasets with consistent improvements.

Abstract

An end-to-end (E2E) ASR model implicitly learns a prior Internal Language Model (ILM) from the training transcripts. To fuse an external LM using Bayes posterior theory, the log likelihood produced by the ILM has to be accurately estimated and subtracted. In this paper we propose two novel approaches to estimate the ILM based on Listen-Attend-Spell (LAS) framework. The first method is to replace the context vector of the LAS decoder at every time step with a vector that is learned with training transcripts. Furthermore, we propose another method that uses a lightweight feed-forward network to directly map query vector to context vector in a dynamic sense. Since the context vectors are learned by minimizing the perplexities on training transcripts, and their estimation is independent of encoder output, hence the ILMs are accurately learned for both methods. Experiments show that the ILMs achieve the lowest perplexity, indicating the efficacy of the proposed methods. In addition, they also significantly outperform the shallow fusion method, as well as two previously proposed ILM Estimation (ILME) approaches on several datasets.