dMel: Speech Tokenization made Simple

TL;DR

This paper introduces dmel, a simple, robust, and training-free speech tokenization method that discretizes mel-filterbank channels, enabling effective speech synthesis and recognition with a unified transformer-based framework.

Contribution

The paper presents a novel discretization of mel-filterbank channels into intensity bins, simplifying speech tokenization and improving robustness and performance over existing methods.

Findings

dmel outperforms existing tokenization methods in preserving audio content

dmel demonstrates robustness to out-of-domain audio signals

RichTTS and RichASR achieve comparable or better results than specialized models

Abstract

Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have investigated various compression-based speech tokenization methods to discretize continuous speech signals, enabling the application of language modeling techniques to discrete tokens. However, audio compressor introduces additional complexity and computational cost, and often fail on out-of-domain audio signals. In this work, we introduce a novel speech representation (dmel) that discretizes mel-filterbank channels into intensity bins, creating a simpler yet more effective representation compared to existing speech tokenization methods. Our approach demonstrates superior performance in preserving audio content, robustness to out-of-domain data, and offers a training-free, natural, and streamable representation. To address the high-dimensional nature of log-mel spectrograms, we propose an efficient parallel encoding and decoding method for high-dimensional tokens using an LM-style transformer architecture. This innovation enables us to develop RichTTS and RichASR, two models sharing the same architecture while achieving comparable or better results than specialized existing methods. Our results demonstrate the effectiveness of dmel in achieving high performance on both speech synthesis and recognition tasks within a unified framework, paving the way for efficient and effective joint modeling of speech and text.