Learning to Count Words in Fluent Speech enables Online Speech Recognition

TL;DR

This paper presents Taris, an online speech recognition system using a Transformer model with an auxiliary incremental word counting task, achieving near offline performance with minimal delay across multiple datasets.

Contribution

Introduction of Taris, a novel online speech recognition system that incorporates incremental word counting to enable eager decoding with minimal delay.

Findings

Online system performs comparably to offline with 5-segment delay

Estimated segment length distribution resembles forced alignment

Negligible overhead compared to standard Transformer

Abstract

Sequence to Sequence models, in particular the Transformer, achieve state of the art results in Automatic Speech Recognition. Practical usage is however limited to cases where full utterance latency is acceptable. In this work we introduce Taris, a Transformer-based online speech recognition system aided by an auxiliary task of incremental word counting. We use the cumulative word sum to dynamically segment speech and enable its eager decoding into words. Experiments performed on the LRS2, LibriSpeech, and Aishell-1 datasets of English and Mandarin speech show that the online system performs comparable with the offline one when having a dynamic algorithmic delay of 5 segments. Furthermore, we show that the estimated segment length distribution resembles the word length distribution obtained with forced alignment, although our system does not require an exact segment-to-word equivalence. Taris introduces a negligible overhead compared to a standard Transformer, while the local relationship modelling between inputs and outputs grants invariance to sequence length by design.