Multi-mode Transformer Transducer with Stochastic Future Context

TL;DR

This paper introduces a Multi-mode ASR model with Stochastic Future Context that dynamically adjusts latency during inference, balancing speed and accuracy without needing multiple models.

Contribution

The paper proposes a novel training method enabling a single ASR model to adapt its latency dynamically, improving flexibility and efficiency over traditional fixed-latency models.

Findings

The model achieves comparable or better accuracy than multiple fixed-latency baselines.

It effectively balances latency and accuracy across different configurations.

Experiments on AISHELL-1 and LibriSpeech validate its versatility.

Abstract

Automatic speech recognition (ASR) models make fewer errors when more surrounding speech information is presented as context. Unfortunately, acquiring a larger future context leads to higher latency. There exists an inevitable trade-off between speed and accuracy. Naively, to fit different latency requirements, people have to store multiple models and pick the best one under the constraints. Instead, a more desirable approach is to have a single model that can dynamically adjust its latency based on different constraints, which we refer to as Multi-mode ASR. A Multi-mode ASR model can fulfill various latency requirements during inference -- when a larger latency becomes acceptable, the model can process longer future context to achieve higher accuracy and when a latency budget is not flexible, the model can be less dependent on future context but still achieve reliable accuracy. In pursuit of Multi-mode ASR, we propose Stochastic Future Context, a simple training procedure that samples one streaming configuration in each iteration. Through extensive experiments on AISHELL-1 and LibriSpeech datasets, we show that a Multi-mode ASR model rivals, if not surpasses, a set of competitive streaming baselines trained with different latency budgets.