Adaptive Knowledge Distillation between Text and Speech Pre-trained Models

TL;DR

This paper introduces PAD, a novel metric-based knowledge distillation method that aligns text and speech models' embedding spaces using adaptive, prior-informed strategies, improving linguistic knowledge transfer without model modifications.

Contribution

It proposes the Prior-informed Adaptive knowledge Distillation (PAD) method, addressing modal disparity and semantic gaps between text and speech models with minimal data and no structural changes.

Findings

PAD outperforms other metric-based distillation methods on spoken language understanding benchmarks.

PAD effectively aligns text and speech embeddings, enhancing linguistic knowledge transfer.

The approach handles variable granularity and prior distributions for better global and local alignment.

Abstract

Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models. With knowledge distillation, these models may also benefit from the knowledge encoded by language models that are pre-trained on rich sources of texts. The distillation process, however, is challenging due to the modal disparity between textual and speech embedding spaces. This paper studies metric-based distillation to align the embedding space of text and speech with only a small amount of data without modifying the model structure. Since the semantic and granularity gap between text and speech has been omitted in literature, which impairs the distillation, we propose the Prior-informed Adaptive knowledge Distillation (PAD) that adaptively leverages text/speech units of variable granularity and prior distributions to achieve better global and local alignments between text and speech pre-trained models. We evaluate on three spoken language understanding benchmarks to show that PAD is more effective in transferring linguistic knowledge than other metric-based distillation approaches.