Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS

TL;DR

This paper investigates the universality of speech discrete tokens generated by SSL models, demonstrating their effectiveness in speech recognition and synthesis, and highlighting their potential to unify multiple speech tasks with lower storage needs.

Contribution

It provides a comprehensive comparison and optimization of discrete tokens across multiple SSL models, showing their potential as universal representations for speech tasks.

Findings

Discrete tokens perform comparably to FBank features in recognition

They outperform mel-spectrograms in speech synthesis

Universal discrete tokens show promise across multiple speech tasks

Abstract

Self-supervised learning (SSL) proficiency in speech-related tasks has driven research into utilizing discrete tokens for speech tasks like recognition and translation, which offer lower storage requirements and great potential to employ natural language processing techniques. However, these studies, mainly single-task focused, faced challenges like overfitting and performance degradation in speech recognition tasks, often at the cost of sacrificing performance in multi-task scenarios. This study presents a comprehensive comparison and optimization of discrete tokens generated by various leading SSL models in speech recognition and synthesis tasks. We aim to explore the universality of speech discrete tokens across multiple speech tasks. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on FBank features in speech recognition tasks and outperform mel-spectrogram features in speech synthesis in subjective and objective metrics. These findings suggest that universal discrete tokens have enormous potential in various speech-related tasks. Our work is open-source and publicly available at https://github.com/k2-fsa/icefall.