SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition

TL;DR

SENS-ASR enhances streaming automatic speech recognition by integrating semantic embeddings derived from past context, significantly reducing word error rates under low-latency conditions.

Contribution

This work introduces a novel semantic embedding injection method into neural transducers for streaming ASR, improving transcription accuracy in low-latency scenarios.

Findings

Significant reduction in Word Error Rate on standard datasets.

Effective semantic information reinforcement improves streaming ASR performance.

Semantic context module trained via knowledge distillation enhances transcription quality.

Abstract

Many Automatic Speech Recognition (ASR) applications require streaming processing of the audio data. In streaming mode, ASR systems need to start transcribing the input stream before it is complete, i.e., the systems have to process a stream of inputs with a limited (or no) future context. Compared to offline mode, this reduction of the future context degrades the performance of Streaming-ASR systems, especially while working with low-latency constraint. In this work, we present SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information. This semantic information is extracted from the available past frame-embeddings by a context module. This module is trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions. Experiments on standard datasets show that SENS-ASR significantly improves the Word Error Rate on small-chunk streaming scenarios.