Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition

TL;DR

This paper evaluates compression-based discrete audio tokens as an alternative to mel-spectrograms for speaker verification, diarization, and speech recognition, showing competitive performance, robustness, and high compression ratios.

Contribution

It provides a comprehensive comparison of compression-based audio tokens with mel-spectrograms across multiple speech tasks, highlighting their potential and limitations.

Findings

Models trained on audio tokens are within 1% of mel-spectrogram performance.

Audio tokens are robust to out-of-domain narrowband data.

Achieves 20x compression with minimal performance loss.

Abstract

Discrete audio representation, aka audio tokenization, has seen renewed interest driven by its potential to facilitate the application of text language modeling approaches in audio domain. To this end, various compression and representation-learning based tokenization schemes have been proposed. However, there is limited investigation into the performance of compression-based audio tokens compared to well-established mel-spectrogram features across various speaker and speech related tasks. In this paper, we evaluate compression based audio tokens on three tasks: Speaker Verification, Diarization and (Multi-lingual) Speech Recognition. Our findings indicate that (i) the models trained on audio tokens perform competitively, on average within $1\%$ of mel-spectrogram features for all the tasks considered, and do not surpass them yet. (ii) these models exhibit robustness for out-of-domain narrowband data, particularly in speaker tasks. (iii) audio tokens allow for compression to 20x compared to mel-spectrogram features with minimal loss of performance in speech and speaker related tasks, which is crucial for low bit-rate applications, and (iv) the examined Residual Vector Quantization (RVQ) based audio tokenizer exhibits a low-pass frequency response characteristic, offering a plausible explanation for the observed results, and providing insight for future tokenizer designs.