Entropy-based Coarse and Compressed Semantic Speech Representation Learning

TL;DR

This paper introduces an entropy-based method for learning compressed semantic speech representations that reduce redundancy and improve efficiency without sacrificing performance in various speech tasks.

Contribution

It proposes a novel entropy-driven aggregation framework that adaptively compresses speech representations, enhancing efficiency and maintaining or improving task performance.

Findings

Compressed representations match or outperform dense tokens in ASR.

Flexible control over granularity and compression ratio.

Effective across multiple speech tasks like translation and voice conversion.

Abstract

Discrete speech representation learning has recently attracted increasing interest in both acoustic and semantic modeling. Existing approaches typically encode 16 kHz waveforms into discrete tokens at a rate of 25 or 50 tokens per second. However, given that speech generally conveys only 2 to 5 words per second, such fine-grained tokenization introduces redundancy and hinders efficiency in downstream training and inference. Moreover, semantic speech representations at this frequency primarily capture phonetic-level information, while semantic understanding may not require such detailed token-level resolution. To address these limitations, we propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations. A speech language model is first pre-trained via next-token prediction on large-scale unlabeled data to capture frequent token patterns. Predictive entropy is then used to adaptively determine aggregation boundaries, followed by a cross-attention module that fuses information within each segment. By adjusting the entropy threshold, the granularity and compression ratio of the representations can be flexibly controlled. Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences, demonstrating the effectiveness of the proposed approach.

Tables

Table 1. Table 1: Performance comparison under different entropy thresholds for ASR, ST, and VC tasks. The entropy thresholds correspond to token rates of 24 Hz (Fine-grained), 15 Hz (Moderately compressed), and 7 Hz (Coarse-grained). ST results are reported on the EN-CN test set.

Threshold (Description)	Token Rate (Hz)	Task	WER (%)	CER (%)	BLEU	Q-MOS	S-MOS	UTMOS
Fine-grained	24	ASR	6.4	3.5	-	-	-	-
		VC	6.7	3.7	-	4.10	3.95	3.98
		ST	-	-	30.1	-	-	-
Moderately compressed	15	ASR	5.6	2.9	-	-	-	-
		VC	7.3	4.2	-	3.82	3.76	3.80
		ST	-	-	31.5	-	-	-
Coarse-grained	7	ASR	6.8	3.6	-	-	-	-
		VC	7.8	4.5	-	3.68	3.58	3.61
		ST	-	-	31.0	-	-	-

Table 2. Table 2: ASR performance at 15 Hz token rate under different entropy-based segmentation criteria: M1 global threshold, M2 relative change, M3 combined.

Method	WER (%)	CER (%)
M1	5.8	3.1
M2	6.3	3.4
M3	5.9	3.2

Table 3. Table 3: ASR performance under different token compression strategies.

Method	WER (%)	CER (%)
HuBERT (original, 50Hz)	6.2	3.3
HuBERT (deduplicated, 26Hz)	5.9	3.1
Entropy-guided (medium, 15Hz)	5.6	2.9
Fixed-length Pooling (2, 25Hz)	6.4	3.5
Entropy-guided (high, 7Hz)	10.5	6.2
Fixed-length Pooling (4, 12.5Hz)	6.8	3.6

Table 4. Table 4: BLEU scores for speech-to-text translation with different compression methods. Higher BLEU score indicates better translation quality.

Method	EN-CN	EN-FR	EN-ES
HuBERT (original, 50 Hz)	28.2	28.5	29.3
HuBERT (deduplicated, 26 Hz)	29.8	30.2	30.7
Entropy-guided (high comp., 7 Hz)	31.0	30.3	30.8
Entropy-guided (medium, 15 Hz)	31.5	30.9	31.6
Fixed-length Pooling (win=2, 25 Hz)	27.3	26.5	28.0
Fixed-length Pooling (win=4, 12.5 Hz)	25.4	24.1	25.6

Table 5. Table 5: Voice conversion results.

Method	Q-MOS	S-MOS	WER (%)
HuBERT (original, 50 Hz)	3.89	3.75	6.2
HuBERT (deduplicated, 26 Hz)	4.12	3.95	6.7
Entropy-guided (medium, 15 Hz)	3.85	3.78	6.4
Entropy-guided (high comp., 7 Hz)	3.33	3.27	9.1
Fixed-length Pooling (win=2, 25 Hz)	3.61	3.52	7.4
Fixed-length Pooling (win=4, 12.5 Hz)	3.25	3.12	9.9

Table 6. Table 6: ASR decoding latency (ms) under different token compression methods.

Compression Method	Latency (ms)
HuBERT (original, 50 Hz)	172
HuBERT (deduplicated, 26 Hz)	148
Fixed-length Pooling (win=2, 25 Hz)	103
Entropy-guided (medium, 15 Hz)	120
Entropy-guided (high compression, 7 Hz)	89

Table 7. Table 7: 15Hz token rate for ASR experiment.

Method	WER	CER
boundary + max pooling (w/o CALE)	6.4	3.4
boundary + average pooling (w/o CALE)	6.8	3.6
boundary + attention pooling (w/o CALE)	6.1	3.3
Ours (with CALE)	5.6	2.9

Table 8. Table 8: 24Hz token rate for voice conversion experiment.

Method	Q-MOS	S-MOS	WER (%)
boundary + max pooling (w/o CALE)	4.02	3.88	7.1
boundary + average pooling (w/o CALE)	3.97	3.85	7.4
boundary + attention pooling (w/o CALE)	4.06	3.92	6.9
Ours (with CALE)	4.10	3.95	6.7

Table 9. Table 9: 15Hz token rate for speech-to-text translation experiment. CALE refers to our Cross-Attentive Local Encoder.

Method	EN-CN	EN-FR	EN-ES
boundary + max pooling (w/o CALE)	30.1	29.4	29.8
boundary + average pooling (w/o CALE)	29.2	28.5	29.1
boundary + attention pooling (w/o CALE)	30.8	30.2	30.7
Ours (with CALE)	31.5	30.9	31.6

Table 10. Table 10: Performance of HuBERT representations (K=2000) on ASR and ST tasks across different token rates.

Threshold (Description)	Token Rate (Hz)	Task	WER (%)	CER (%)	BLEU
Fine-grained	24	ASR	4.5	2.7	-
		ST	-	-	31.7
Moderately compressed	15	ASR	3.9	2.4	-
		ST	-	-	33.2
Coarse-grained	7	ASR	5.8	3.1	-
		ST	-	-	32.1

Table 11. Table 11: Boundary alignment analysis at different compression rates. CR: Compression Rate; ASD: Avg Span Duration; WBA: Word Boundary Alignment; PBA: Phoneme Boundary Alignment; MTD: Mean Time Deviation.

CR (Hz)	ASD (ms)	WBA (%)	PBA (%)	MTD (ms)
Original (50Hz)	20	-	-	-
Deduplicated (26Hz)	38	11.4	32.7	42.3
24Hz	42	18.9	35.3	38.7
15Hz	67	22.6	83.2	15.8
7Hz	143	89.7	28.1	28.6

Table 12. Table 12: ASR results with various pooling methods under the fixed-length setting.

Method	WER (%)	CER (%)
Entropy-guided (medium, 15Hz)	5.6	2.9
Fixed-length max pooling (2, 25Hz)	6.4	3.5
Fixed-length average pooling (2, 25Hz)	6.7	3.8
Fixed-length attention pooling (2, 25Hz)	6.2	3.4
Fixed-length max pooling (4, 12.5Hz)	6.8	3.6
Fixed-length average pooling (4, 12.5Hz)	7.2	3.9
Fixed-length attention pooling (4, 12.5Hz)	6.6	3.5

Table 13. Table 13: Comparison of boundary identification methods across different granularities. Note: M1 = global threshold ( θ g \theta_{g} ), M2 = relative change ( θ r \theta_{r} ), M3 = combined ( θ g + θ r \theta_{g}+\theta_{r} ). For metrics, ↓ \downarrow indicates lower is better, and ↑ \uparrow indicates higher is better.

Token Rate	Method	ASR (WER $\downarrow$)	ST (BLEU $\uparrow$)
24Hz	M1 (${\theta }_g=0.78$)	6.4	30.1
	M2 (${\theta }_r=0.65$)	7.0	29.4
	M3 (${\theta }_g=0.75,{\theta }_r=0.62$)	6.3	30.3
15Hz	M1 (${\theta }_g=0.85$)	5.8	31.5
	M2 (${\theta }_r=0.69$)	6.3	30.2
	M3 (${\theta }_g=0.82,{\theta }_r=0.65$)	5.9	31.2
7Hz	M1 (${\theta }_g=0.97$)	6.8	31.0
	M2 (${\theta }_r=0.71$)	7.5	29.8
	M3 (${\theta }_g=0.86,{\theta }_r=0.68$)	7.0	31.2

Equations

$$\mathbf{u}=\{u_1,u_2,\dots ,u_N\},u_i\in \{1,\dots ,K\}.$$

$$p(u_i\mid u_{<i})\text{for }{u}_i\in \{1,\dots ,K\},$$

$$H(u_i)=-\sum _{v=1}^{K}p(u_i=v\mid u_{<i})\log p(u_i=v\mid u_{<i}),$$

$$H(u_i)>{\theta }_g,$$

$$H(u_i)-H(u_{i-1})>{\theta }_r,$$

$$g_j=\{u_{b_{j-1}+1},\dots ,u_{b_j}\},j=1,\dots ,M.$$

$${\mathbf{p}}_j^{(0)}=\mathrm{MaxPool}\left(\left\{{\mathbf{e}}_i\mid i\in g_j\right\}\right).$$

$${\mathbf{q}}_j^{(\ell )}$$

$${\mathbf{k}}_i^{(\ell )}$$

$${\mathbf{v}}_i^{(\ell )}$$

$${\alpha }_{j,i}^{(\ell )}$$

$${\mathbf{z}}_j^{(\ell )}$$

$${\mathbf{p}}_j^{(\ell )}$$

Full text

**Entropy-based Coarse and Compressed Semantic

Speech Representation Learning**

** Jialong Zuo1, Guangyan Zhang2, Minghui Fang1, Shengpeng Ji1, Xiaoqi Jiao2**

Jingyu Li2, Yiwen Guo3, Zhou Zhao1††thanks: Corresponding author.

1Zhejiang University  2LIGHTSPEED  3Independent Researcher

jialongzuo@zju.edu.cn   zhaozhou@zju.edu.cn

1 Introduction

In recent years, discrete representation learning of speech Défossez et al. (2022); Kumar et al. (2023); Zhang et al. (2023); Ji et al. (2024) has achieved remarkable success and has gradually emerged as a crucial bridge between the speech modality and large language models Xie and Wu (2024a, b); Xu et al. (2025). A notable direction within this paradigm is semantic-level discretization, aimed at retaining only the high-level semantic content of audio signals, enabling much lower bitrates. For tasks that primarily rely on speech content such as automatic speech recognition (ASR) or speech-to-text translation, semantic tokens provide a more efficient and task-relevant representation.

Self-supervised learning (SSL) of speech Schneider et al. (2019); Baevski et al. (2020); Hsu et al. (2021); Chung et al. (2021); Chen et al. (2022) has demonstrated remarkable success in capturing rich and transferable speech representations across a wide range of downstream tasks. Building on this foundation, a growing number of studies have sought to extract semantic units from SSL-derived speech features. HuBERT Hsu et al. (2021) represents a pioneering effort in this direction, employing k-means clustering to tokenize speech, where the resulting discrete units are used as training targets for the speech encoder and are found to exhibit strong phoneme-level correlations Choi et al. (2024). Subsequent works have adopted similar approaches to derive semantic representations: Spirit-LM Nguyen et al. (2025) extracts HuBERT features and applies k-means clustering with 500 centroids to define its basic semantic units, while Vevo Zhang et al. (2025) follows a comparable methodology. SpeechTokenizer Zhang et al. (2023) leverages semantic representations extracted from a HuBERT-based teacher model to guide the distillation of the first layer of residual vector quantization (RVQ), thereby generating discrete units intended to capture semantic content. Similarly, AudioLM Borsos et al. (2023) utilizes semantic tokens derived from w2v-BERT Chung et al. (2021) to encode high-level semantic information from audio inputs.

In addition to extracting semantic units from self-supervised speech representations, several methods employ supervised approaches such as ASR or phoneme recognition to obtain semantic tokens. For instance, CosyVoice Du et al. (2024) proposes to use a supervised ASR module to construct a supervised semantic speech (S3) tokenizer. FACodec Ju et al. (2024) directly leverages phoneme labels (specifically, frame-level phoneme annotations obtained from internal alignment tools) to train discrete content representations in a supervised manner. While the effectiveness of these semantic tokenizers in capturing high-level semantic units has been validated across various downstream tasks such as ASR Yang et al. (2023), text-to-speech (TTS) Du et al. (2024); Wang et al. (2024), and voice conversion (VC) Kim et al. (2023); Liu (2024); Zuo et al. (2025), several critical limitations remain to be addressed. First, existing semantic discretization approaches typically compress speech into 25 or more tokens per second, meaning that long speech utterances often result in an excessive number of discrete units. This introduces a substantial computational burden for training and inference in downstream models. Second, the semantic tokens often encode excessive phonetic detail Sicherman and Adi (2023); Cho et al. (2023), which may be unnecessarily fine-grained for tasks focused on semantic understanding. Therefore, both further compression and coarser-grained semantic representations are necessary and desirable.

In this study, we propose an entropy-based semantic token compression strategy for dynamically extracting semantic speech representations at varying levels of granularity. Specifically, building upon HuBERT units (discretized via a K-Means clustering method), we achieve further sequence compression by training a lightweight auto-regressive language model (LM) on large-scale pretrained speech data and computing the entropy of the next-token distribution. By leveraging the LM’s ability to model frequently occurring patterns (i.e., sequences of adjacent tokens), we define an entropy threshold to segment the token stream and merge adjacent tokens accordingly. This approach effectively reduces the sequence length. Furthermore, we demonstrate that modifying the entropy threshold and segmentation criteria enables flexible control over compression granularity, which subsequently influence the performance of downstream semantic understanding and generation tasks. The contributions of this work are summarized as follows:

• •

We propose an entropy-based token aggregation framework that adaptively compresses semantic speech representations by merging adjacent tokens based on predictive uncertainty derived from a lightweight autoregressive language model.

• •

Our method enables controllable granularity in semantic token sequences, significantly reducing sequence length while retaining essential semantic content, thus enhancing efficiency for downstream speech understanding and generation tasks.

• •

Empirical results across multiple benchmarks demonstrate that our approach achieves a favorable balance between compression rate and task performance, validating its effectiveness and scalability.

2 Related Work

2.1 Self-supervised Speech Representation Learning

Self-supervised learning (SSL) has emerged as a powerful paradigm for extracting rich and generalizable representations from large-scale unlabeled speech corpora. Recent advances such as wav2vec Baevski et al. (2020), HuBERT Hsu et al. (2021), and WavLM Chen et al. (2022) have demonstrated the effectiveness of SSL in capturing both phonetic and semantic information from raw audio signals. These models typically rely on pretext tasks such as masked prediction or contrastive learning to model the temporal structure of speech without requiring manual annotations. For example, wav2vec employs a contrastive loss to distinguish true audio segments from distractors, thereby learning high-level representations. HuBERT adopts a masked prediction objective, where targets are derived from offline k-means clustering, encouraging the model to learn context-aware representations. W2v-BERT Chung et al. (2021) further combines contrastive and masked learning in an end-to-end framework to leverage the benefits of both objectives. Meanwhile, WavLM further introduces a denoising objective to enhance robustness against noise and speaker variation.

These models typically operate at a fixed frame rate (e.g., 50 Hz), producing fine-grained representations that align closely with sub-phonemic units Choi et al. (2024). While effective for phonetic modeling, such granularity can be redundant for tasks requiring only high-level semantic understanding, motivating the need for more compact and task-relevant representations.

2.2 Extracting Semantic Units from Speech

Semantic units, or semantic tokens, are discrete representations derived from speech that encapsulate higher-level phonetic or linguistic information. These units are typically extracted either from self-supervised learning (SSL) models or through supervised training paradigms.

In SSL-based approaches, semantic tokens are often obtained by quantizing hidden representations using methods such as K-means or VQ-VAE van den Oord et al. (2018). This process, applied to pretrained models like HuBERT or Wav2Vec 2.0, yields discrete indices that correlate with phonetic or semantic properties. Some SSL models incorporate internal quantizers, trained jointly with the backbone network, producing token-like outputs that serve as training targets. While conventionally termed "semantic", recent findings suggest that these tokens may primarily encode phonetic structure rather than higher-level semantics Cho et al. (2024, 2023); Choi et al. (2024). Nevertheless, they have demonstrated strong performance across a range of downstream tasks.

Supervised methods, by contrast, offer a more explicit mechanism for semantic token extraction. Models such as the S³ Tokenizer Du et al. (2024) integrate vector quantization within a Transformer-based architecture and are trained with ASR-like objectives on large-scale paired speech-text corpora. This allows for more preservation of paralinguistic information than directly transcribing speech into text. These supervised tokenizers are trained on massive paired speech-text data, and have demonstrated rich speech content understanding capabilities.

3 Method

3.1 Overall Architecture

The overall architecture of the proposed entropy-based semantic compression framework is illustrated in Figure 1. It consists of three main components: (1) Discrete Semantic Token Extraction: input speech is first processed by a pretrained HuBERT model, and the resulting continuous representations are discretized using k-means clustering to obtain frame-level semantic tokens. (2) Entropy-Based Grouping Module: a lightweight autoregressive language model is trained on these tokens to model their distribution and compute token-wise conditional entropy. Based on a designed entropy threshold, adjacent tokens with low predictive uncertainty are merged into variable-length groups, enabling sequence compression. (3) Cross-Attentive Local Encoder: the compressed token groups are fed into a local encoder equipped with a cross-attention mechanism, enabling efficient incorporation of compressed representations into downstream understanding and generation tasks. The framework allows flexible control over the compression ratio by adjusting the entropy threshold, supporting different granularity levels for diverse applications.

3.2 Discrete Semantic Token Extraction

An input waveform is first encoded by a pretrained HuBERT model and discretized via k-means clustering into frame-level semantic tokens with a codebook size of $K$:

[TABLE]

These tokens serve as the discrete input units for subsequent entropy-based compression model.

3.3 Entropy-Based Grouping Module

The primary objective of the Entropy-Based Grouping Module is to adaptively compress the token sequence $\mathbf{u}$ by leveraging entropy as a measure of uncertainty in the token predictions. To achieve this, we train a lightweight autoregressive language model (LM) on the discrete token sequence. The LM predicts the probability distribution of each token $u_{i}$ conditioned on its preceding tokens, thereby capturing the sequential dependencies and uncertainties in the token sequence. Formally, for each position $i$, the LM estimates the conditional distribution:

[TABLE]

where $u_{i}$ represents the current token and $u_{<i}$ denotes the sequence of preceding tokens. This distribution is essential for quantifying the uncertainty associated with each token prediction.

Token-wise Entropy. To measure the uncertainty of each token prediction, we compute the token-wise entropy $H(u_{i})$, which is based on the predictive distribution $p(u_{i}\mid u_{<i})$. This entropy quantifies the unpredictability or uncertainty in the token’s prediction:

[TABLE]

where $H(u_{i})$ is the entropy of the predicted distribution over all possible tokens $v\in\{1,\dots,K\}$. A higher entropy value indicates greater uncertainty in the prediction, while a lower entropy suggests a more confident prediction. This entropy-based measure enables an adaptive segmentation strategy, where boundaries are determined by detecting sharp increases in conditional entropy, allowing segments of low uncertainty to be merged while capturing transitions in the underlying linguistic organization.

Boundary Identification. We introduce two entropy-based criteria to identify the boundaries that separate different groups of tokens. These criteria rely on both the token’s individual entropy and the relative change in entropy between consecutive tokens. The first criterion uses a global entropy threshold $\theta_{g}$ to identify regions with high uncertainty:

[TABLE]

This global threshold $\theta_{g}$ is empirically determined and ensures that tokens with sufficiently high entropy are considered as potential group boundaries. The second criterion captures abrupt changes in uncertainty between adjacent tokens by examining the relative difference in entropy between $u_{i}$ and $u_{i-1}$:

[TABLE]

where $\theta_{r}$ is a relative threshold that emphasizes significant increases in uncertainty. By setting this criterion, we can identify boundaries that reflect changes in the predictability of the token sequence, which likely correspond to boundaries at the phonetic or sub-phonemic level, consistent with the characteristics of HuBERT tokens.

Group Formation. Once the boundary conditions are applied, we identify the positions $b_{0},b_{1},\dots,b_{M}$ where the boundaries occur, with $b_{0}=0$ and $b_{M}=N$, where $N$ is the total number of tokens in the sequence. These indices define the boundaries of $M$ contiguous token groups:

[TABLE]

Each group $g_{j}$ consists of tokens between two consecutive boundary indices. This adaptive grouping process dynamically adjusts the length of each group based on local token predictability. In particular, the groups formed in regions with high uncertainty will likely be shorter, while groups in more predictable regions will be longer. The final result is a compressed sequence $\{g_{j}\}_{j=1}^{M}$, where each group represents a semantically coherent subset of the original token sequence. This compression helps reduce the overall sequence length while maintaining key semantic information.

By tuning the thresholds $\theta_{g}$ and $\theta_{r}$, the Entropy-Based Grouping Module offers fine-grained control over the trade-off between compression ratio and the preservation of semantic structure. This adaptive grouping process plays a critical role in the overall compression framework, ensuring that high-entropy regions are appropriately compressed while minimizing the loss of important semantic content.

3.4 Cross-Attentive Local Encoder

The cross-attentive local encoder aims to derive compact group-level representations $\{\mathbf{p}_{j}\}$ from token sequences grouped into segments $\{g_{j}\}$. Unlike standard pooling approaches, our design iteratively refines group embeddings through stacked cross-attention layers, allowing information aggregation from the constituent tokens in each group.

Token Embedding. Each discrete token $u_{i}$ is embedded into a continuous vector $\mathbf{e}_{i}\in\mathbb{R}^{d}$ via a learnable embedding matrix $\mathbf{W}_{e}\in\mathbb{R}^{K\times d}$, where $K$ is the vocabulary size and $d$ is the embedding dimension. The input token sequence is thus transformed into $\{\mathbf{e}_{i}\}_{i=1}^{N}$.

Initialization of Group Queries. For each group $g_{j}=\{u_{b_{j-1}+1},\dots,u_{b_{j}}\}$, we initialize its representation $\mathbf{p}_{j}^{(0)}\in\mathbb{R}^{d}$ by applying max pooling over the token embeddings:

[TABLE]

Cross-Attention Update. We employ $L$ cross-attention layers to iteratively refine the group representations. At the $\ell$-th layer ($\ell=1,\dots,L$), the group representation $\mathbf{p}_{j}^{(\ell)}$ is updated via attention over the token representations $\{\mathbf{h}_{i}^{(\ell-1)}\}$, where $\mathbf{h}_{i}^{(0)}=\mathbf{e}_{i}$. The update is defined as:

[TABLE]

After $L$ cross-attention layers, the final group embedding is defined as $\mathbf{p}_{j}=\mathbf{p}_{j}^{(L)}$. Attention is inherently restricted to tokens within each group by their index range. At each layer, the query $\mathbf{q}_{j}^{(\ell)}$ is recomputed from the previous group embedding, facilitating hierarchical and localized aggregation of token information. Residual connections and pre-layer normalization are adopted for stable optimization. We omit positional encodings, since intra-group order is implicitly captured through attention weights.

Group-Level Output Representation. The resulting sequence of group embeddings $\{\mathbf{p}_{j}\}_{j=1}^{M}$ is subsequently passed to downstream modules for further semantic understanding or generation tasks. By adjusting the parameters $\theta_{g}$ and $\theta_{r}$, the framework provides fine-grained control over the trade-off between the compression ratio and the preservation of semantic accuracy.

4 Experiments

4.1 Experiment Setup

Datasets We utilize the English portion of the Multilingual LibriSpeech (MLS) dataset Pratap et al. (2020), comprising approximately 20,000 hours of speech, to pretrain the Entropy LLM and to provide training data for both the ASR and voice conversion tasks. For ASR and zero-shot voice conversion evaluation, we use the standard test-clean subset from LibriSpeech Panayotov et al. (2015), covering a diverse set of speakers. To assess speech-to-text translation performance, we adopt the benchmark CVSS-C dataset Jia et al. (2022), which is derived from the CoVoST 2 speech-to-text translation corpus Wang et al. (2021). We conduct experiments on three language pairs: English-Chinese (EN-CN), English-Spanish (EN-ES), and English-French (EN-FR) covering a range of widely used languages with varying linguistic characteristics.

4.2 Training and Inference Setup

Entropy LLM Pretraining The Entropy LLM follows a lightweight LLaMA-style Touvron et al. (2023) architecture with approximately 0.1 billion parameters. It consists of 8 Transformer layers with a model dimension of 1024, 16 attention heads, and a maximum sequence length of 2048. Rotary positional embeddings (RoPE) Su et al. (2023) are used, and all hidden dimensions are aligned to multiples of 256. The model is trained for 750k steps on the English subset of MLS (20k hours) using the AdamW optimizer with a learning rate of $3\times 10^{-4}$, weight decay of 0.01, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-5}$. A cosine learning rate schedule with 2k warmup steps and a final learning rate ratio of 0.1 is applied. We utilize the open-source pretrained HuBERT model111https://github.com/facebookresearch/fairseq/blob/main/examples/hubert and a $k$-means clustering model for semantic unit extraction, with the number of clusters set to $k=500$. For the high compression setting at 7 Hz token rate, the global entropy threshold is $\theta_{g}=9.7$.

ASR and Speech-to-Text Translation For ASR and speech translation tasks, we adopt an encoder-decoder architecture similar to Whisper Radford et al. (2022). The encoder is configured either as a standard Conformer encoder Gulati et al. (2020) for raw semantic units, or as a lightweight local encoder tailored for compressed coarse semantic speech representations. The decoder generates output text autoregressively. Speech translation is evaluated on the CVSS-C benchmark, covering English–Chinese, English–Spanish, and English–French pairs.

Voice Conversion The voice conversion model is based on a Diffusion Transformer similar to Liu (2024). It takes as input either original semantic tokens or compressed semantic representations, along with a reference mel-spectrogram. The output mel-spectrogram is synthesized using a pretrained HiFi-GAN vocoder Kong et al. (2020).

4.3 Evaluation Metrics

We evaluate ASR and voice conversion using both objective and subjective metrics. ASR performance is assessed by Word Error Rate (WER) and Character Error Rate (CER), while voice conversion quality is evaluated via Q-MOS (naturalness and clarity) and S-MOS (speaker similarity) through human ratings on Amazon Mechanical Turk, complemented by automatic predictions using UTMOS Saeki et al. (2022). For speech-to-text translation, we report BLEU scores computed between the generated and reference text. Inference latency is measured as the average decoding time per input utterance, evaluated on a single NVIDIA V100 GPU over the test set.

5 Exploration

We conduct a series of experiments to examine how entropy-based token compression and the granularity of the compressed representations affect performance across various speech-related tasks. Our study centers on two key aspects:

(1) Entropy-Based Semantic Compression with Varying Granularity. We systematically explore how different entropy thresholds and boundary selection strategies, applied to the entropy-based language model, influence the granularity of compressed semantic representations. By modulating these parameters, we obtain semantic units of varying granularity. These compressed representations are then evaluated across multiple speech understanding and generation tasks, including ASR, speech translation, and voice conversion, offering empirical insights into the selection of optimal compression levels tailored to the specific characteristics and demands of each task.

(2) Evaluation of Semantic Token Compression Methods. We investigate the performance of four distinct semantic token compression approaches across multiple downstream tasks, providing a comprehensive assessment. These approaches include (i) original HuBERT tokens, (ii) deduplicated HuBERT tokens, (iii) fixed-length downsampled pooling tokens, and (iv) entropy-based compressed tokens at varying compression levels. This comparison highlights the advantages of our entropy-based compression method relative to alternative strategies.

5.1 Experiment on Granularity Modulation Using Entropy Thresholds

This experiment investigates the effect of varying entropy thresholds on the granularity of semantic tokens and their consequent impact on performance in ASR, speech-to-text translation (ST), and voice conversion (VC) tasks. The entropy threshold regulates the segmentation process and the extent of compression applied to the speech representations, resulting in token sequences with varying granularity levels. Specifically, three entropy thresholds are examined, yielding token compression rates of approximately 24 Hz for fine-grained tokens, 15 Hz for moderately compressed tokens, and 7 Hz for coarse-grained tokens. The performance metrics for each task under different entropy thresholds are summarized in Table 1.

For ASR and speech-to-text translation tasks, moderate compression under the medium entropy threshold yields the best performance among the evaluated settings, as it aligns well with typical phoneme rates of around 17–18 Hz. Fine-grained tokens provide more detailed phonetic representations but may introduce redundancy and increase computational overhead. In contrast, coarse-grained tokens reduce token density at the cost of losing certain useful linguistic cues, which compromises recognition performance relative to the moderately compressed configuration.

Conversely, for the generative voice conversion task, semantic tokens that avoid being overly coarse in granularity, such as token rates below 10 Hz, tend to yield better performance in Q-MOS, S-MOS, and UTMOS. While coarser token representations may slightly improve S-MOS, they often reduce Q-MOS and intelligibility. Therefore, maintaining sufficiently fine-grained semantic token representations is important to capture the subtle acoustic nuances necessary for high-quality voice conversion. These findings underscore the necessity of adapting entropy-based compression parameters to the specific requirements of downstream tasks. Recognition-oriented applications such as ASR and ST benefit from a balanced compression strategy, whereas generative tasks like voice conversion require finer semantic granularity to ensure fidelity and naturalness in speech synthesis.

6 Entropy-Based Grouping Criteria

We investigate the impact of two entropy-based grouping criteria on ASR performance under moderate token compression (15 Hz). The first method employs a global entropy threshold $\theta_{g}$ to identify tokens with high uncertainty as group boundaries. The second method detects abrupt changes in entropy by examining the relative difference between consecutive tokens, controlled by a relative threshold $\theta_{r}$. Additionally, we test a combined approach that applies both criteria simultaneously. As shown in Table 2, Method 1, which uses the global entropy threshold alone, achieves the best ASR performance, with a WER of 5.8% and a CER of 3.1%. Method 2, relying solely on relative entropy changes, results in slightly degraded accuracy. The combined approach provides marginal improvement over Method 2 but does not surpass the effectiveness of Method 1. These results suggest that the global entropy threshold is sufficient to segment semantic tokens into meaningful groups for ASR under moderate compression. Consequently, we adopt Method 1 as the default grouping strategy in subsequent experiments to maintain a balance between simplicity and performance.

6.1 ASR Performance

We evaluate the impact of various token compression strategies on ASR performance, measured by WER and CER. As shown in Table 3, the Entropy-guided (medium) configuration (15 Hz), which approximates the phoneme rate, achieves the best recognition performance, slightly outperforming both HuBERT (deduplicated) (26 Hz) and HuBERT (original) (50 Hz). This suggests that moderate entropy-based compression effectively preserves essential linguistic information while reducing redundancy. The Fixed-length pooling approach with a window size of 2 (25 Hz) yields comparable results, but increasing the window to 4 (12.5 Hz) leads to a notable drop in accuracy, likely due to excessive smoothing that suppresses important fine-grained speech content details. Notably, the Entropy-guided (high comp.) setting (7 Hz), despite its lower token rate, the entropy-guided method outperforms the fixed-length pooling strategy at 12.5 Hz, demonstrating the effectiveness of entropy-based compression in preserving essential linguistic representations under substantial compression.

6.2 Speech-to-Text Translation

Table 4 shows that Entropy-guided compression at 15 Hz achieves the best BLEU scores across all language pairs, indicating effective preservation of semantic content. The high compression variant (7 Hz) performs worse but still outperforms HuBERT deduplicated (26 Hz), suggesting that entropy-based selection better maintains key linguistic units under stronger compression. In contrast, Fixed-length Pooling leads to significant BLEU degradation, especially with larger pooling windows, due to the loss of important semantic and contextual details. Compared to ASR, speech-to-text translation is less dependent on fine-grained acoustic or phonetic information. The relative robustness of ST performance under coarser token compression suggests that moderate to coarse granularity sufficiently preserves the linguistic content needed for accurate translation.

6.3 Voice Conversion

Table 5 shows that performance degrades progressively with increased compression. The original 50 Hz and deduplicated 26 Hz tokens yield the best results, while excessive reduction especially via fixed-length pooling significantly harms naturalness and intelligibility. Entropy-guided compression at 15 Hz maintains competitive quality, suggesting it can moderately reduce token rates without severely affecting performance. These results indicate that voice conversion relies more heavily on detailed acoustic and prosodic information, and is less tolerant to coarse compression than tasks like speech translation.

6.4 Decoding Latency

Decoding latency decreases as token compression increases, with high compression yielding the fastest inference but reduced accuracy. Moderate compression achieves a favorable balance, significantly lowering latency while maintaining competitive performance. This highlights the importance of selecting appropriate compression levels to optimize ASR systems for efficiency.

7 Conclusion

This work presents an entropy-based dynamic aggregation framework for compressing semantic speech representations by leveraging predictive uncertainty from a lightweight autoregressive language model. The proposed method effectively reduces token sequence length while preserving critical semantic information, enabling controllable granularity and improving computational efficiency. Experimental results on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations achieve comparable or superior performance to dense token sequences. These findings highlight the potential of entropy-guided compression as a flexible and scalable approach for efficient semantic speech modeling in diverse downstream applications.

8 Limitations and Future Work

Although the proposed entropy-based semantic representation compression framework demonstrates superior performance in semantic understanding tasks, enabling adjustable granularity according to task requirements, it exhibits limitations in generation-related tasks. Specifically, the quality of generated outputs tends to degrade as the compression ratio monotonically increases, suggesting that this approach may be less suitable for compressing sequences of acoustic tokens. Furthermore, additional experiments on other types of semantic tokens are necessary to validate the generalizability and broader applicability of the method. Future work will focus on addressing these limitations and exploring extensions to enhance the effectiveness of entropy-guided compression across a wider range of speech representation tasks.

Appendix A Detailed Experiment Settings

A.1 Details in Subjective Evaluation

We randomly select 50 sentences from the test set and perform the subjective evaluation on Amazon Mechanical Turk (MTurk). Each generated audio has been listened to by at least 10 native listeners. For Q-MOS evaluations, the listeners are instructed to focus on assessing the audio quality and naturalness while disregarding any differences in styles (such as timbre, emotion, and pronunciation). Conversely, for S-MOS evaluations, the listeners are instructed to concentrate on evaluating the speaker similarity to the audio prompt, while disregarding differences in content or audio quality. For the Q-MOS, S-MOS evaluations, each listener is asked to rate different speech samples using a Likert scale ranging from 1 to 5.

Appendix B Details of Models

An open-source HiFi-GAN vocoder222https://www.modelscope.cn/models/iic/CosyVoice-300M is used to synthesize speech from mel-spectrograms. In the cross-attentive local encoder, we employ 4 layers of Transformer blocks. To obtain semantic representations at different levels of granularity, we adjust the entropy threshold $\theta_{g}$ within the range of 7.5 to 10.5. Specifically, for token rates of 7 Hz, 15 Hz, and 24 Hz, the thresholds are set to $\theta_{g}=9.7$, $8.5$, and $7.8$, respectively.

For the ASR task, we adopt an autoregressive decoder architecture similar to Whisper, utilizing an encoder-decoder structure with a total of approximately 104M parameters to generate text from compressed token sequences. For voice conversion (VC), we follow the design of Seed-VC Liu (2024) and use a Diffusion Transformer model as the generator. The compressed semantic tokens are aligned with the mel-spectrogram in time by padding zero tensors, and combined with mel-spectrogram features as additional conditions. These are then fed into a flow matching model to generate the final mel-spectrograms.

Appendix C Additional Experiments

C.1 About Cross-Attentive Local Encoder

To further clarify the effectiveness of our Cross-Attentive Local Encoder (CALE) module, we conducted additional ablation studies across three tasks. The results are summarized below:

From the above ablation study results, we can observe that when only using the corresponding grouping boundaries + pooling methods for aggregation without incorporating the Cross-Attentive Local Encoder, our method shows varying degrees of performance degradation across all three tasks. We analyze that beyond helping with intra-group information aggregation, the introduction of the Cross-Attentive Local Encoder module enables the model to achieve global perception of the original input, supplementing missing global information. This global perception capability allows the model to better understand the contextual relationships both within and across semantic boundaries, thereby enhancing overall performance.

C.2 About the K-means clusters

Due to computational resource and time constraints, we conducted additional experiments with HuBERT representations using K=2000 clusters to evaluate the performance of our compression framework across different token rates on ASR and ST tasks. The results are presented in the table 10.

Compared to our original experiments with K=500, the results with K=2000 demonstrate consistent improvements across both ASR and ST tasks. Most importantly, our core finding remains valid: the dynamic compression framework effectively identifies optimal compression rates for different tasks, with moderately compressed representations (15 Hz) achieving the best efficiency-performance trade-off.

Regarding alternative SSL features like WavLM, we acknowledge that both HuBERT and WavLM encode high-level semantic representations. We believe our compression framework would naturally transfer to these models, though resource constraints prevented comprehensive evaluation. These additional experiments with K=2000 strengthen our analysis while confirming the robustness of our proposed framework across different parameter settings.

C.3 About semantic structure

We also conducted boundary alignment analysis comparing our entropy-based compressed token boundaries with linguistically meaningful boundaries obtained through the open-source Montreal Forced Aligner (MFA) tool 333https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner on the test-clean subset.

Metrics : Word/Phoneme Boundary Alignment measures the percentage of compressed boundaries that fall within ±50ms of MFA-aligned boundaries. Mean Time Deviation represents the average absolute temporal distance between compressed and reference boundaries.

According to the table, our entropy-based compression method demonstrates clear frequency-boundary correspondence relationships. The 15Hz compression rate achieves optimal phoneme boundary alignment (83.2%), with its 67ms average span duration closely matching phoneme-level linguistic units. The 7Hz high compression rate reaches the highest performance in word boundary alignment (89.7%), with its 143ms span duration corresponding to typical word unit durations. The 26Hz deduplication processing, with its 38ms span duration falling between phoneme and word units, shows moderate performance on both boundary alignments. This systematic correspondence pattern demonstrates that our compressed tokens capture linguistically meaningful structures at different granularities, with "semantic structure" referring to the hierarchical linguistic units preserved at each compression level.

C.4 About the fixed-length downsampling

In our original experiments, the fixed-length downsampling refers to using a fixed-sized window to aggregate the representations within that window, where we applied max pooling as the default aggregation method. Since different pooling strategies may affect performance to some extent, we conducted additional ASR experiments with various pooling methods under the fixed-length setting. The results are shown in the table below:

The results show that while different pooling strategies under fixed-length settings lead to slightly varied performance, none surpass our entropy-guided method. This confirms that the gain comes from adaptive boundary modeling rather than pooling choice alone. The performance gaps also become more pronounced at lower token rates (e.g., 12.5Hz), highlighting the importance of boundary quality.

C.5 Ablation on Entropy Thresholds

In Table 1, we present the performance of Boundary Identification Method 1 (global entropy threshold $\theta_{g}$) across three tasks at different compression granularities: 24Hz with $\theta_{g}=0.78$, 15Hz with $\theta_{g}=0.85$, and 7Hz with $\theta_{g}=0.97$. In Table 2, we focus on 15Hz (the optimal token rate for ASR) and compare three segmentation strategies: $\theta_{g}$ only (M1), $\theta_{r}$ only (M2), and Combined ($\theta_{g}+\theta_{r}$, M3). Results show that M1 and M3 consistently outperform M2, and their performance is close. To keep the experimental setup simple and efficient, we adopt the $\theta_{g}$-only method in the main experiments for boundary identification. Internally, we also tested the $\theta_{g}$-only approach across a wider range of token rates (6Hz–30Hz) on all three tasks and selected 24Hz, 15Hz, and 7Hz as representative settings.

We acknowledge that further exploration of $\theta_{g}$ and $\theta_{r}$ combinations is meaningful, but it requires substantial computational resources. Based on Table 2, we now provide extended results comparing all three methods (M1, M2, M3) under different granularities on ASR (WER $\downarrow$) and ST (BLEU, EN-CN $\uparrow$):

Observed the above table, we can draw several important conclusions that extend our findings from Table 2. While our previous analysis focused on the 15Hz setting (optimal for ASR tasks), this comprehensive evaluation across multiple token rates (24Hz, 15Hz, 7Hz) provides stronger evidence for our boundary identification approach. The consistent performance pattern M1$\approx$M3$>$M2 across different compression granularities reinforces our decision to adopt the single global threshold $\theta_{g}$ method for subsequent experiments. Notably, the performance gaps remain relatively stable across token rates, with M1 and M3 consistently achieving lower WER (5.8-6.8) and higher BLEU scores (30.1-31.5) compared to M2, which demonstrates that the segmentation methods of M1 and M3 are more reasonable. Through more extensive ablation studies on $\theta_{g}$ and $\theta_{r}$ combinations, we observe that the global entropy threshold is sufficient to segment semantic tokens into meaningful groups.