DuoTok: Source-Aware Dual-Track Tokenization for Multi-Track Music Language Modeling

TL;DR

DuoTok introduces a source-aware dual-track tokenizer for multi-track music modeling, balancing high-fidelity reconstruction and predictability, and demonstrating improved performance on standard benchmarks.

Contribution

It presents a staged disentanglement approach for dual-track tokenization, enhancing predictability and fidelity in multi-track music language models.

Findings

Achieves lowest cnBPT on benchmarks while maintaining 0.75 kbps reconstruction.

Improves enBPT under dual-track language modeling protocol.

Models trained on DuoTok tokens utilize cross-track structure and long-range context.

Abstract

Audio tokenization bridges continuous waveforms and multi-track music language models. In dual-track modeling, tokens should preserve three properties at once: high-fidelity reconstruction, strong predictability under a language model, and cross-track correspondence. We introduce DuoTok, a source-aware dual-track tokenizer that addresses this trade-off through staged disentanglement. DuoTok first pretrains a semantic encoder, then regularizes it with multi-task supervision, freezes the encoder, and applies hard dual-codebook routing while keeping auxiliary objectives on quantized codes. A diffusion decoder reconstructs high-frequency details, allowing tokens to focus on structured information for sequence modeling. On standard benchmarks, DuoTok achieves a favorable predictability-fidelity trade-off, reaching the lowest cnBPT while maintaining competitive reconstruction at 0.75 kbps. Under a held-constant dual-track language modeling protocol, enBPT also improves, indicating gains beyond codebook size effects. Controlled diagnostics show larger predictability costs under cross-track corruption and larger gains from longer context, suggesting that models trained on DuoTok tokens use cross-track structure and non-local history.