MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates

TL;DR

MSR-HuBERT introduces a multi-sampling-rate adaptive pre-training method that enables speech models to effectively handle mixed sampling rate data without resampling, improving performance in speech recognition and reconstruction.

Contribution

It proposes a novel multi-sampling-rate adaptive downsampling CNN integrated into HuBERT, allowing unified training on mixed-rate speech data without resampling.

Findings

Outperforms HuBERT on speech recognition tasks across 16-48 kHz.

Preserves high-frequency details in speech reconstruction.

Enables effective mixed-rate pre-training and fine-tuning.

Abstract

Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.