A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations

TL;DR

This paper investigates how self-supervised speech models encode speaker-specific information, revealing a hierarchy of features across layers and challenging assumptions about final layer representations, with implications for interpretability.

Contribution

It provides a large-scale probing analysis of 11 speech SSL models, uncovering how speaker attributes are encoded across layers and highlighting unexpected patterns in larger models.

Findings

Fundamental acoustics are encoded in initial layers.

Abstract traits are synthesized in middle layers.

Larger models recover speaker identity in deep layers.

Abstract

Enhancing explainability in speech self-supervised learning (SSL) is important for developing reliable SSL-based speech processing systems. This study probes how speech SSL models encode speaker-specific information via a large-scale probing analysis of 11 models, decomposing identity into acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. Crucially, the consensus that final layers purely abstract linguistic content is challenged. It is discovered that larger models unexpectedly recover speaker identity in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings. These insights decode the complex internal mechanics of SSL models, providing guidelines for selecting interpretable and task-optimal representations.