WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation

TL;DR

WavSLM introduces a novel single-stream speech language model trained via WavLM distillation, enabling coherent speech generation without text supervision and supporting streaming inference with fewer resources.

Contribution

It presents a new approach to speech modeling that combines semantic and acoustic information in a single stream through distillation, simplifying architecture and training.

Findings

Achieves competitive results on consistency benchmarks

Supports streaming inference efficiently

Uses fewer parameters and less training data

Abstract

Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to speech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference. Demo samples are available at https://lucadellalib.github.io/wavslm-web/.