Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification

TL;DR

This paper introduces a knowledge distillation approach from BERT to a speech transformer model for intent classification, improving accuracy and robustness in speech understanding tasks without relying on large speech datasets.

Contribution

It presents a novel multilevel transformer-based knowledge distillation method from BERT to speech models for intent classification, enhancing performance in low-resource and noisy conditions.

Findings

Achieved 99.10% accuracy on Fluent speech corpus.

Achieved 88.79% accuracy on ATIS database.

Demonstrated improved robustness in acoustically degraded environments.

Abstract

End-to-end intent classification using speech has numerous advantages compared to the conventional pipeline approach using automatic speech recognition (ASR), followed by natural language processing modules. It attempts to predict intent from speech without using an intermediate ASR module. However, such end-to-end framework suffers from the unavailability of large speech resources with higher acoustic variation in spoken language understanding. In this work, we exploit the scope of the transformer distillation method that is specifically designed for knowledge distillation from a transformer based language model to a transformer based speech model. In this regard, we leverage the reliable and widely used bidirectional encoder representations from transformers (BERT) model as a language model and transfer the knowledge to build an acoustic model for intent classification using the speech. In particular, a multilevel transformer based teacher-student model is designed, and knowledge distillation is performed across attention and hidden sub-layers of different transformer layers of the student and teacher models. We achieve an intent classification accuracy of 99.10% and 88.79% for Fluent speech corpus and ATIS database, respectively. Further, the proposed method demonstrates better performance and robustness in acoustically degraded condition compared to the baseline method.