DASB - Discrete Audio and Speech Benchmark

TL;DR

DASB introduces a comprehensive benchmarking framework for discrete audio tokens across multiple domains, revealing their current limitations and guiding future improvements.

Contribution

The paper presents DASB, a standardized benchmark for evaluating discrete audio tokens, addressing inconsistencies and providing insights into their robustness and performance.

Findings

Discrete representations are less robust than continuous ones.

Semantic tokens outperform acoustic tokens but still lag behind continuous features.

Careful tuning of model factors is essential for optimal performance.

Abstract

Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling multimodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.