SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models

TL;DR

SECodec introduces a novel speech representation method based on structural entropy, improving speech encoding efficiency and quality for speech language models, and addresses limitations of existing Euclidean distance-based quantization.

Contribution

The paper proposes SECodec, a structural entropy-based speech codec that models speech as a graph and hierarchically minimizes 2D SE, offering adaptive quantization and improved speech representation.

Findings

Performs comparably to EnCodec in speech reconstruction.

Surpasses VALL-E in zero-shot text-to-speech tasks.

Provides a new information-theoretic approach to speech discretization.

Abstract

With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretization rely on a predefined codebook size and Euclidean distance-based quantization. However, 1) the size of codebook is a critical parameter that affects both codec performance and downstream task training efficiency. 2) The Euclidean distance-based quantization may lead to audio distortion when the size of the codebook is controlled within a reasonable range. In fact, in the field of information compression, structural information and entropy guidance are crucial, but previous methods have largely overlooked these factors. Therefore, we address the above issues from an information-theoretic perspective, we present SECodec, a novel speech representation codec based on structural entropy (SE) for building speech language models. Specifically, we first model speech as a graph, clustering the speech features nodes within the graph and extracting the corresponding codebook by hierarchically and disentangledly minimizing 2D SE. Then, to address the issue of audio distortion, we propose a new quantization method. This method still adheres to the 2D SE minimization principle, adaptively selecting the most suitable token corresponding to the cluster for each incoming original speech node. Furthermore, we develop a Structural Entropy-based Speech Language Model (SESLM) that leverages SECodec. Experimental results demonstrate that SECodec performs comparably to EnCodec in speech reconstruction, and SESLM surpasses VALL-E in zero-shot text-to-speech tasks. Code, demo speeches, speech feature graph, SE codebook, and models are available at https://github.com/wlq2019/SECodec.