VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction

TL;DR

VocalNet-M2 introduces a low-latency spoken language model that uses multi-codebook tokenization and multi-token prediction to significantly reduce response delay while maintaining high performance for real-time speech applications.

Contribution

It presents a novel integrated multi-codebook tokenizer and multi-token prediction strategy that reduces latency and improves efficiency in end-to-end spoken language models.

Findings

Reduced first chunk latency from 725ms to 350ms

Maintained competitive performance with mainstream SLMs

Provided insights into multi-codebook strategies for real-time applications

Abstract

Current end-to-end spoken language models (SLMs) have made notable progress, yet they still encounter considerable response latency. This delay primarily arises from the autoregressive generation of speech tokens and the reliance on complex flow-matching models for speech synthesis. To overcome this, we introduce VocalNet-M2, a novel low-latency SLM that integrates a multi-codebook tokenizer and a multi-token prediction (MTP) strategy. Our model directly generates multi-codebook speech tokens, thus eliminating the need for a latency-inducing flow-matching model. Furthermore, our MTP strategy enhances generation efficiency and improves overall performance. Extensive experiments demonstrate that VocalNet-M2 achieves a substantial reduction in first chunk latency (from approximately 725ms to 350ms) while maintaining competitive performance across mainstream SLMs. This work also provides a comprehensive comparison of single-codebook and multi-codebook strategies, offering valuable insights for developing efficient and high-performance SLMs for real-time interactive applications.