Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERT

TL;DR

This paper introduces a novel self-supervised method for syllable discovery that disentangles speaker information from linguistic content in speech representations, improving syllable segmentation quality.

Contribution

It proposes a speaker-disentangled fine-tuning approach for HuBERT that enhances syllabic unit extraction without relying on transcriptions.

Findings

Outperforms state-of-the-art in syllable segmentation metrics

Effective separation of speaker and linguistic information

Improves syllabic organization in speech representations

Abstract

Self-supervised speech representation learning has become essential for extracting meaningful features from untranscribed audio. Recent advances highlight the potential of deriving discrete symbols from the features correlated with linguistic units, which enables text-less training across diverse tasks. In particular, sentence-level Self-Distillation of the pretrained HuBERT (SD-HuBERT) induces syllabic structures within latent speech frame representations extracted from an intermediate Transformer layer. In SD-HuBERT, sentence-level representation is accumulated from speech frame features through self-attention layers using a special CLS token. However, we observe that the information aggregated in the CLS token correlates more with speaker identity than with linguistic content. To address this, we propose a speech-only self-supervised fine-tuning approach that separates syllabic units from speaker information. Our method introduces speaker perturbation as data augmentation and adopts a frame-level training objective to prevent the CLS token from aggregating paralinguistic information. Experimental results show that our approach surpasses the current state-of-the-art method in most syllable segmentation and syllabic unit quality metrics on Librispeech, underscoring its effectiveness in promoting syllabic organization within speech-only models.