Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data

TL;DR

This paper introduces a joint speech and text optimization method for hybrid transducer and attention-based encoder decoder models, improving ASR accuracy and enabling effective domain adaptation without additional speech data.

Contribution

It proposes the J-TAED model that leverages both speech and text data during training, unifies multi-modal representations, and enhances domain adaptation capabilities.

Findings

Reduces WER by 5.8% to 12.8% on Librispeech.

Achieves 15.3% and 17.8% WER reduction on out-of-domain datasets.

Enables domain adaptation without requiring speech data.

Abstract

A joint speech and text optimization method is proposed for hybrid transducer and attention-based encoder decoder (TAED) modeling to leverage large amounts of text corpus and enhance ASR accuracy. The joint TAED (J-TAED) is trained with both speech and text input modalities together, while it only takes speech data as input during inference. The trained model can unify the internal representations from different modalities, and be further extended to text-based domain adaptation. It can effectively alleviate data scarcity for mismatch domain tasks since no speech data is required. Our experiments show J-TAED successfully integrates speech and linguistic information into one model, and reduce the WER by 5.8 ~12.8% on the Librispeech dataset. The model is also evaluated on two out-of-domain datasets: one is finance and another is named entity focused. The text-based domain adaptation brings 15.3% and 17.8% WER reduction on those two datasets respectively.