Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute

TL;DR

This paper demonstrates how to optimize HuBERT self-supervised speech models to be trained efficiently on limited academic resources, achieving comparable performance to large-scale models with significantly fewer GPUs.

Contribution

The authors reproduce and optimize HuBERT training for resource-constrained environments, enabling effective SSL training with only 8 GPUs and introducing semi-supervised pre-training methods.

Findings

Models trained with our methods match original HuBERT performance.

Training time and resource requirements are significantly reduced.

Open-source code and models are provided for community use.

Abstract

Self-supervised learning (SSL) has led to great strides in speech processing. However, the resources needed to train these models has become prohibitively large as they continue to scale. Currently, only a few groups with substantial resources are capable of creating SSL models, which harms reproducibility. In this work, we optimize HuBERT SSL to fit in academic constraints. We reproduce HuBERT independently from the original implementation, with no performance loss. Our code and training optimizations make SSL feasible with only 8 GPUs, instead of the 32 used in the original work. We also explore a semi-supervised route, using an ASR model to skip the first pre-training iteration. Within one iteration of pre-training, our models improve over HuBERT on several tasks. Furthermore, our HuBERT Large variant requires only 8 GPUs, achieving similar performance to the original trained on 128. As our contribution to the community, all models, configurations, and code are made open-source in ESPnet.