JOOCI: a Framework for Learning Comprehensive Speech Representations

TL;DR

JOOCI is a new speech representation learning framework that effectively captures both content and paralinguistic information simultaneously, outperforming existing models on multiple benchmark tasks.

Contribution

It introduces a novel method that jointly optimizes for content and other speech information without layer-wise trade-offs, improving performance over prior SSL models.

Findings

JOOCI outperforms WavLM by 26.5% on benchmark tasks.

It achieves superior results on speaker recognition and language tasks.

The method effectively captures both content and paralinguistic features.

Abstract

Information in speech can be categorized into two groups: Content (what is being said, such as linguistics) and Other (how it is expressed such as information about speaker and paralinguistic features). Current self-supervised learning (SSL) methods are shown to divide the model's representational-depth or layers in two, with earlier layers specializing in Other and later layers in Content related tasks. This layer-wise division is inherently sub-optimal, as neither information type can use all layers to build hierarchical representations. To address this, we propose JOOCI, a novel speech representation learning method that does not compromise on the representational-depth for either information type. JOOCI outperforms WavLM by 26.5%, and other models of similar size (100M parameters), when evaluated on two speaker recognition and two language tasks from the SUPERB benchmark, demonstrating its effectiveness in Jointly Optimizing Other and Content Information (JOOCI).