CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders

TL;DR

This paper introduces CoLLD, a novel contrastive layer-to-layer distillation method that effectively compresses large multilingual speech encoders, enabling efficient deployment without significant performance loss.

Contribution

The paper proposes a new knowledge distillation technique combining masked prediction and contrastive learning to improve compression of large speech models.

Findings

CoLLD outperforms prior compression methods.

It closes the performance gap on multilingual speech recognition and translation.

Effective for on-device speech applications.

Abstract

Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studies propose model compression methods to address this issue, but those works focus on smaller models and less realistic tasks. Thus, we propose Contrastive Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to compress pre-trained speech encoders by leveraging masked prediction and contrastive learning to train student models to copy the behavior of a large teacher model. CoLLD outperforms prior methods and closes the gap between small and large models on multilingual speech-to-text translation and recognition benchmarks.