Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition

TL;DR

This paper introduces a memory-augmented lookup dictionary for Transformer language models that improves long-tail token prediction in speech recognition without sacrificing decoding efficiency.

Contribution

It proposes a novel memory-augmented lookup dictionary architecture that enhances long-tail token prediction in language models for ASR.

Findings

Significant reduction in word/character error rates on Chinese and English datasets.

Improved tail token error rates compared to baseline Transformer LMs.

No impact on decoding efficiency.

Abstract

Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequently in the training set is still quite challenging. The long-tail prediction problems have been widely studied in many applications, but only been addressed by a few studies for ASR and LMs. In this paper, we propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. With intensive experiments on Chinese and English data sets, our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate. This is achieved without impact on the decoding efficiency. Overall, we demonstrate the effectiveness of our proposed method in boosting the ASR decoding performance, especially for long-tail tokens.