Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study

TL;DR

This paper systematically explores the use of discrete speech units derived from self-supervised learning to improve efficiency and performance in speech recognition, translation, and understanding tasks, demonstrating promising results across multiple datasets.

Contribution

It provides a comprehensive evaluation of discrete speech units in end-to-end models, showing their effectiveness and efficiency improvements in various speech processing tasks.

Findings

Discrete units achieve good results across tasks

Training time is significantly reduced

Configurations and models will be publicly released

Abstract

Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts.