Cross-Utterance Language Models with Acoustic Error Sampling

TL;DR

This paper introduces a cross-utterance language model that leverages surrounding utterances and acoustic error sampling to improve speech recognition accuracy, demonstrating significant WER reductions on multiple datasets.

Contribution

It proposes a novel cross-utterance language model with an extraction network and acoustic error sampling to enhance ASR performance.

Findings

Outperforms baseline LSTM language models in WER

Achieves up to 0.9% absolute WER reduction on Eval2000 datasets

Demonstrates effectiveness on AMI and Switchboard datasets

Abstract

The effective exploitation of richer contextual information in language models (LMs) is a long-standing research problem for automatic speech recognition (ASR). A cross-utterance LM (CULM) is proposed in this paper, which augments the input to a standard long short-term memory (LSTM) LM with a context vector derived from past and future utterances using an extraction network. The extraction network uses another LSTM to encode surrounding utterances into vectors which are integrated into a context vector using either a projection of LSTM final hidden states, or a multi-head self-attentive layer. In addition, an acoustic error sampling technique is proposed to reduce the mismatch between training and test-time. This is achieved by considering possible ASR errors into the model training procedure, and can therefore improve the word error rate (WER). Experiments performed on both AMI and Switchboard datasets show that CULMs outperform the LSTM LM baseline WER. In particular, the CULM with a self-attentive layer-based extraction network and acoustic error sampling achieves 0.6% absolute WER reduction on AMI, 0.3% WER reduction on the Switchboard part and 0.9% WER reduction on the Callhome part of Eval2000 test set over the respective baselines.