Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism

TL;DR

This paper enhances Transformer-based conversational speech recognition by explicitly modeling inter-sentential context, using a context-aware residual attention mechanism and a conditional decoder, leading to improved performance across multiple dialogue datasets.

Contribution

It introduces a novel inter-sentential attention mechanism and a conditional decoder for Transformer-based ASR, explicitly capturing cross-utterance context in conversational speech.

Findings

Consistent performance improvements on open-source dialogue corpora

Effective modeling of cross-utterance context enhances recognition accuracy

Proposed method outperforms utterance-level Transformer models

Abstract

Transformer-based models have demonstrated their effectiveness in automatic speech recognition (ASR) tasks and even shown superior performance over the conventional hybrid framework. The main idea of Transformers is to capture the long-range global context within an utterance by self-attention layers. However, for scenarios like conversational speech, such utterance-level modeling will neglect contextual dependencies that span across utterances. In this paper, we propose to explicitly model the inter-sentential information in a Transformer based end-to-end architecture for conversational speech recognition. Specifically, for the encoder network, we capture the contexts of previous speech and incorporate such historic information into current input by a context-aware residual attention mechanism. For the decoder, the prediction of current utterance is also conditioned on the historic linguistic information through a conditional decoder framework. We show the effectiveness of our proposed method on several open-source dialogue corpora and the proposed method consistently improved the performance from the utterance-level Transformer-based ASR models.