What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study

TL;DR

This paper systematically studies speech tokenizer designs in speech-language models, introducing multi-token prediction and speaker-aware generation to improve speech synthesis quality, decoding speed, and speaker consistency.

Contribution

It compares different speech tokenizer configurations, introduces multi-token prediction for faster decoding, and proposes a speaker-aware paradigm with a new QA benchmark.

Findings

Decoupled tokenization improves alignment and synthesis quality.

Multi-token prediction speeds up decoding by up to 12× and reduces word error rate.

Speaker-aware generation enhances speaker consistency and knowledge understanding.

Abstract

Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation. However, challenges remain in achieving effective cross-modal alignment and high-quality speech generation. In this work, we systematically investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling. We compare coupled, semi-decoupled, and fully decoupled speech tokenizers under a fair SLM framework and find that decoupled tokenization significantly improves alignment and synthesis quality. To address the information density mismatch between speech and text, we introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens. This leads to up to 12$\times$ faster decoding and a substantial drop in word error rate (from 6.07 to 3.01). Furthermore, we propose a speaker-aware generation paradigm and introduce RoleTriviaQA, a large-scale role-playing knowledge QA benchmark with diverse speaker identities. Experiments demonstrate that our methods enhance both knowledge understanding and speaker consistency.