Beyond the Utterance: An Empirical Study of Very Long Context Speech Recognition

TL;DR

This study demonstrates that modern attention-based speech recognition models can effectively process very long audio sequences, significantly improving performance by leveraging extended context up to an hour.

Contribution

The paper shows that current hardware and algorithms enable training ASR models on sequences over an hour long, surpassing traditional short-utterance limitations.

Findings

Using up to 21.8 minutes of context improves accuracy by 14.2%.

Encoding positional information and model architecture are crucial for long-sequence performance.

Models utilize both linguistic and acoustic information from distant context.

Abstract

Automatic speech recognition (ASR) models are normally trained to operate over single utterances, with a short duration of less than 30 seconds. This choice has been made in part due to computational constraints, but also reflects a common, but often inaccurate, modelling assumption that treats utterances as independent and identically distributed samples. When long-format audio recordings are available, to work with such systems, these recordings must first be segmented into short utterances and processed independently. In this work, we show that due to recent algorithmic and hardware advances, this is no longer necessary, and current attention-based approaches can be used to train ASR systems that operate on sequences of over an hour in length. Therefore, to gain a better understanding of the relationship between the training/evaluation sequence length and performance, we train ASR models on large-scale data using 10 different sequence lengths from 10 seconds up to 1 hour. The results show a benefit from using up to 21.8 minutes of context, with up to a 14.2% relative improvement from a short context baseline in our primary experiments. Through modifying various architectural components, we find that the method of encoding positional information and the model's width/depth are important factors when working with long sequences. Finally, a series of evaluations using synthetic data are constructed to help analyse the model's use of context. From these results, it is clear that both linguistic and acoustic aspects of the distant context are being used by the model.