How Should We Extract Discrete Audio Tokens from Self-Supervised Models?

TL;DR

This paper investigates optimal methods for extracting discrete audio tokens from self-supervised models, proposing a scalable universal vocoder and attention mechanism to improve token quality across various audio tasks.

Contribution

It introduces a scalable training approach for a universal vocoder and employs attention to identify task-specific SSL layers, enhancing semantic token extraction.

Findings

Universal vocoder enables consistent tokenization across SSL layers

Attention mechanism improves task-specific layer selection

Enhanced performance in diverse audio processing tasks

Abstract

Discrete audio tokens have recently gained attention for their potential to bridge the gap between audio and language processing. Ideal audio tokens must preserve content, paralinguistic elements, speaker identity, and many other audio details. Current audio tokenization methods fall into two categories: Semantic tokens, acquired through quantization of Self-Supervised Learning (SSL) models, and Neural compression-based tokens (codecs). Although previous studies have benchmarked codec models to identify optimal configurations, the ideal setup for quantizing pretrained SSL models remains unclear. This paper explores the optimal configuration of semantic tokens across discriminative and generative tasks. We propose a scalable solution to train a universal vocoder across multiple SSL layers. Furthermore, an attention mechanism is employed to identify task-specific influential layers, enhancing the adaptability and performance of semantic tokens in diverse audio applications.