Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

TL;DR

This paper introduces a unified post-training framework for speech models that can generate multiple utterance-level attribute representations, improving performance on multilingual speech retrieval and speaker recognition tasks.

Contribution

The work extends speech foundation models to learn multiple utterance-level attributes simultaneously within a single unified framework.

Findings

Jointly learned semantic and speaker representations improve task performance.

The approach enhances multilingual speech retrieval accuracy.

Speaker recognition benefits from the unified attribute representations.

Abstract

Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XSLR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech-text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level. In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.