Text-Utilization for Encoder-dominated Speech Recognition Models

TL;DR

This paper explores efficient methods to leverage text-only data in encoder-dominated speech recognition models, demonstrating that simple configurations can outperform complex ones and improve performance.

Contribution

It introduces practical techniques for integrating text data into encoder-focused speech recognition models, simplifying training and enhancing accuracy.

Findings

Larger encoders with smaller decoders can match or outperform larger decoder architectures.

Simple configurations like random duration models are often more effective than complex methods.

All code and recipes are publicly available for reproducibility.

Abstract

This paper investigates efficient methods for utilizing text-only data to improve speech recognition, focusing on encoder-dominated models that facilitate faster recognition. We provide a comprehensive comparison of techniques to integrate text-only data, including modality matching and dynamic downsampling to reach text-level representations within the encoder. Our experiments on the LibriSpeech corpus show that a larger encoder with a smaller decoder can equal or surpass the performance of architectures with larger decoders. We demonstrate that simple configurations, such as random duration models, are often more effective than complex alternatives, significantly simplifying the training pipeline. All code and recipes are made publicly available.