Context Dependent RNNLM for Automatic Transcription of Conversations

TL;DR

This paper introduces a novel RNN-based language model that incorporates larger conversational context through learned embeddings and gating mechanisms, significantly improving speech recognition accuracy.

Contribution

The paper proposes a new architecture for conversational language modeling that effectively models long-term dependencies across multiple utterances using context embeddings and gating networks.

Findings

Significant perplexity reduction over baseline RNNLMs.

Improved ASR rescoring results with lower WER on both datasets.

Effective modeling of macro-topic context in conversations.

Abstract

Conversational speech, while being unstructured at an utterance level, typically has a macro topic which provides larger context spanning multiple utterances. The current language models in speech recognition systems using recurrent neural networks (RNNLM) rely mainly on the local context and exclude the larger context. In order to model the long term dependencies of words across multiple sentences, we propose a novel architecture where the words from prior utterances are converted to an embedding. The relevance of these embeddings for the prediction of next word in the current sentence is found using a gating network. The relevance weighted context embedding vector is combined in the language model to improve the next word prediction, and the entire model including the context embedding and the relevance weighting layers is jointly learned for a conversational language modeling task. Experiments are performed on two conversational datasets - AMI corpus and the Switchboard corpus. In these tasks, we illustrate that the proposed approach yields significant improvements in language model perplexity over the RNNLM baseline. In addition, the use of proposed conversational LM for ASR rescoring results in absolute WER reduction of $1.2$\% on Switchboard dataset and $1.0$\% on AMI dataset over the RNNLM based ASR baseline.