Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-End

TL;DR

This paper introduces a novel approach to improve speech recognition by incorporating explicit intent representations, derived from an audio-to-intent model, into RNN-T based ASR systems, resulting in notable WER reductions.

Contribution

It is the first to integrate intent embeddings as auxiliary inputs in RNN-T ASR systems, enhancing accuracy especially in streaming scenarios.

Findings

Non-streaming intent-based system reduces WER by 5.56%.

Streaming intent-based system reduces WER by 3.33%.

Significant improvements on media-related intents, e.g., 9.12% WERR on PlayMusicIntent.

Abstract

Comprehending the overall intent of an utterance helps a listener recognize the individual words spoken. Inspired by this fact, we perform a novel study of the impact of explicitly incorporating intent representations as additional information to improve a recurrent neural network-transducer (RNN-T) based automatic speech recognition (ASR) system. An audio-to-intent (A2I) model encodes the intent of the utterance in the form of embeddings or posteriors, and these are used as auxiliary inputs for RNN-T training and inference. Experimenting with a 50k-hour far-field English speech corpus, this study shows that when running the system in non-streaming mode, where intent representation is extracted from the entire utterance and then used to bias streaming RNN-T search from the start, it provides a 5.56% relative word error rate reduction (WERR). On the other hand, a streaming system using per-frame intent posteriors as extra inputs for the RNN-T ASR system yields a 3.33% relative WERR. A further detailed analysis of the streaming system indicates that our proposed method brings especially good gain on media-playing related intents (e.g. 9.12% relative WERR on PlayMusicIntent).