JOIST: A Joint Speech and Text Streaming Model For ASR

TL;DR

JOIST introduces a novel streaming encoder end-to-end model trained jointly on speech-text paired data and text-only unpaired data, improving speech recognition accuracy while maintaining real-time streaming capabilities.

Contribution

It is the first to explore joint training with both speech and text modalities in a streaming E2E model, using significantly larger datasets and different text modeling techniques.

Findings

Text modeling improves WER by 4-14% relative.

JOIST maintains streaming capabilities.

Joint training with unpaired text data enhances performance.

Abstract

We present JOIST, an algorithm to train a streaming, cascaded, encoder end-to-end (E2E) model with both speech-text paired inputs, and text-only unpaired inputs. Unlike previous works, we explore joint training with both modalities, rather than pre-training and fine-tuning. In addition, we explore JOIST using a streaming E2E model with an order of magnitude more data, which are also novelties compared to previous works. Through a series of ablation studies, we explore different types of text modeling, including how to model the length of the text sequence and the appropriate text sub-word unit representation. We find that best text representation for JOIST improves WER across a variety of search and rare-word test sets by 4-14% relative, compared to a model not trained with text. In addition, we quantitatively show that JOIST maintains streaming capabilities, which is important for good user-level experience.