Adapting Long Context NLM for ASR Rescoring in Conversational Agents

TL;DR

This paper enhances neural language models for conversational agent speech recognition by integrating turn-based context, metadata, and user speech patterns, leading to improved accuracy in ASR rescoring and downstream tasks.

Contribution

It introduces novel techniques for incorporating turn-based context and metadata into LSTM and Transformer-XL models, and adapts models to user speech patterns for better ASR performance.

Findings

Achieved 1.6% to 9.1% relative WER reduction.

Improved slot labeling F1 score by 4%.

Enhanced contextual NLMs with attention and fusion methods.

Abstract

Neural Language Models (NLM), when trained and evaluated with context spanning multiple utterances, have been shown to consistently outperform both conventional n-gram language models and NLMs that use limited context. In this paper, we investigate various techniques to incorporate turn based context history into both recurrent (LSTM) and Transformer-XL based NLMs. For recurrent based NLMs, we explore context carry over mechanism and feature based augmentation, where we incorporate other forms of contextual information such as bot response and system dialogue acts as classified by a Natural Language Understanding (NLU) model. To mitigate the sharp nearby, fuzzy far away problem with contextual NLM, we propose the use of attention layer over lexical metadata to improve feature based augmentation. Additionally, we adapt our contextual NLM towards user provided on-the-fly speech patterns by leveraging encodings from a large pre-trained masked language model and performing fusion with a Transformer-XL based NLM. We test our proposed models using N-best rescoring of ASR hypotheses of task-oriented dialogues and also evaluate on downstream NLU tasks such as intent classification and slot labeling. The best performing model shows a relative WER between 1.6% and 9.1% and a slot labeling F1 score improvement of 4% over non-contextual baselines.