SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

TL;DR

SPEAR introduces a unified self-supervised learning framework that effectively combines speech and general audio representations into a single model, outperforming existing models on multiple benchmarks.

Contribution

SPEAR is the first framework to distill knowledge from separate speech and audio teachers into one unified model using multi-codebook vector quantisation and novel training strategies.

Findings

Outperforms existing models on SUPERB benchmark

Achieves state-of-the-art on 12 of 15 SUPERB tasks

Shows competitive performance on HEAR benchmark

Abstract

Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released.