Towards the Next Frontier in Speech Representation Learning Using Disentanglement

TL;DR

This paper introduces Learn2Diss, a novel framework for disentangling speaker and phonemic information in speech representations using joint frame-level and utterance-level encoders, improving various downstream tasks.

Contribution

It proposes a new disentanglement framework combining frame and utterance-level encoders with mutual information criteria for speech representation learning.

Findings

Achieves state-of-the-art results on multiple speech tasks.

Frame-level representations enhance semantic understanding.

Utterance-level representations improve non-semantic tasks.

Abstract

The popular frameworks for self-supervised learning of speech representations have largely focused on frame-level masked prediction of speech regions. While this has shown promising downstream task performance for speech recognition and related tasks, this has largely ignored factors of speech that are encoded at coarser level, like characteristics of the speaker or channel that remain consistent through-out a speech utterance. In this work, we propose a framework for Learning Disentangled Self Supervised (termed as Learn2Diss) representations of speech, which consists of frame-level and an utterance-level encoder modules. The two encoders are initially learned independently, where the frame-level model is largely inspired by existing self supervision techniques, thereby learning pseudo-phonemic representations, while the utterance-level encoder is inspired by constrastive learning of pooled embeddings, thereby learning pseudo-speaker representations. The joint learning of these two modules consists of disentangling the two encoders using a mutual information based criterion. With several downstream evaluation experiments, we show that the proposed Learn2Diss achieves state-of-the-art results on a variety of tasks, with the frame-level encoder representations improving semantic tasks, while the utterance-level representations improve non-semantic tasks.