Improved Factorized Neural Transducer Model For text-only Domain Adaptation

TL;DR

This paper introduces an improved factorized neural Transducer (IFNT) model that better integrates acoustic and language information, enhancing domain adaptation and reducing error rates in end-to-end speech recognition for English and Mandarin.

Contribution

The paper presents the IFNT model, which effectively combines acoustic and language data, outperforming previous FNT and neural Transducer models in domain adaptation tasks.

Findings

IFNT surpasses baseline neural Transducer performance.

Achieves up to 30.2% relative WER reduction on out-of-domain data.

Demonstrates significant accuracy improvements on source domains.

Abstract

Adapting End-to-End ASR models to out-of-domain datasets with text data is challenging. Factorized neural Transducer (FNT) aims to address this issue by introducing a separate vocabulary decoder to predict the vocabulary. Nonetheless, this approach has limitations in fusing acoustic and language information seamlessly. Moreover, a degradation in word error rate (WER) on the general test sets was also observed, leading to doubts about its overall performance. In response to this challenge, we present the improved factorized neural Transducer (IFNT) model structure designed to comprehensively integrate acoustic and language information while enabling effective text adaptation. We assess the performance of our proposed method on English and Mandarin datasets. The results indicate that IFNT not only surpasses the neural Transducer and FNT in baseline performance in both scenarios but also exhibits superior adaptation ability compared to FNT. On source domains, IFNT demonstrated statistically significant accuracy improvements, achieving a relative enhancement of 1.2% to 2.8% in baseline accuracy compared to the neural Transducer. On out-of-domain datasets, IFNT shows relative WER(CER) improvements of up to 30.2% over the standard neural Transducer with shallow fusion, and relative WER(CER) reductions ranging from 1.1% to 2.8% on test sets compared to the FNT model.