Sylber: Syllabic Embedding Representation of Speech from Raw Audio

TL;DR

Sylber introduces a self-supervised learning framework that produces structured syllabic speech representations, enabling efficient tokenization, robust segmentation, and effective speech compression suitable for language modeling.

Contribution

The paper presents a novel SSL method for syllabic speech embedding that achieves fast, robust segmentation and efficient tokenization without domain tuning.

Findings

Achieves an average of 4.27 tokens per second for syllable segmentation

Enables speech reconstruction with lower bitrate than baseline SSL tokens

Categorical perception naturally emerges in Sylber embeddings

Abstract

Syllables are compositional units of spoken language that efficiently structure human speech perception and production. However, current neural speech representations lack such structure, resulting in dense token sequences that are costly to process. To bridge this gap, we propose a new model, Sylber, that produces speech representations with clean and robust syllabic structure. Specifically, we propose a self-supervised learning (SSL) framework that bootstraps syllabic embeddings by distilling from its own initial unsupervised syllabic segmentation. This results in a highly structured representation of speech features, offering three key benefits: 1) a fast, linear-time syllable segmentation algorithm, 2) efficient syllabic tokenization with an average of 4.27 tokens per second, and 3) novel phonological units suited for efficient spoken language modeling. Our proposed segmentation method is highly robust and generalizes to out-of-domain data and unseen languages without any tuning. By training token-to-speech generative models, fully intelligible speech can be reconstructed from Sylber tokens with a significantly lower bitrate than baseline SSL tokens. This suggests that our model effectively compresses speech into a compact sequence of tokens with minimal information loss. Lastly, we demonstrate that categorical perception-a linguistic phenomenon in speech perception-emerges naturally in Sylber, making the embedding space more categorical and sparse than previous speech features and thus supporting the high efficiency of our tokenization. Together, we present a novel SSL approach for representing speech as syllables, with significant potential for efficient speech tokenization and spoken language modeling.