Word Level Timestamp Generation for Automatic Speech Recognition and Translation

TL;DR

This paper presents a data-driven method for enabling word-level timestamp prediction directly within an end-to-end speech recognition model, improving downstream task performance without external alignment modules.

Contribution

The authors introduce a novel <|timestamp|> token in the Canary model, allowing direct timestamp prediction and extending the approach to speech translation tasks.

Findings

Timestamp prediction accuracy between 80-90%.

Error rates range from 20 to 120 ms across four languages.

Minimal impact on word error rate (WER).

Abstract

We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model. Accurate timestamp information is crucial for a variety of downstream tasks such as speech content retrieval and timed subtitles. While traditional hybrid systems and end-to-end (E2E) models may employ external modules for timestamp prediction, our approach eliminates the need for separate alignment mechanisms. By leveraging the NeMo Forced Aligner (NFA) as a teacher model, we generate word-level timestamps and train the Canary model to predict timestamps directly. We introduce a new <|timestamp|> token, enabling the Canary model to predict start and end timestamps for each word. Our method demonstrates precision and recall rates between 80% and 90%, with timestamp prediction errors ranging from 20 to 120 ms across four languages, with minimal WER degradation. Additionally, we extend our system to automatic speech translation (AST) tasks, achieving timestamp prediction errors around 200 milliseconds.