Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

TL;DR

This paper introduces a two-stage textual knowledge distillation approach for end-to-end spoken language understanding, leveraging rich speech features and data augmentation to achieve state-of-the-art accuracy, especially in low-resource scenarios.

Contribution

It proposes a novel two-stage knowledge distillation method matching utterance-level representations and logits, improving SLU performance with a new speech encoder and data augmentation techniques.

Findings

Achieved 99.7% accuracy on Fluent Speech Commands dataset.

Demonstrated effectiveness of data augmentation in low-resource settings.

Validated the importance of each component through ablation studies.

Abstract

End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or fine-tuning with knowledge distillation. To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterance-level representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially. We use vq-wav2vec BERT as a speech encoder because it captures general and rich features. Furthermore, we improve the performance, especially in a low-resource scenario, with data augmentation methods by randomly masking spans of discrete audio tokens and contextualized hidden representations. Consequently, we push the state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy in the full dataset setting and 99.5% in the 10% subset setting. Throughout the ablation studies, we empirically verify that all used methods are crucial to the final performance, providing the best practice for spoken language understanding. Code is available at https://github.com/clovaai/textual-kd-slu.