Adaptive Knowledge Distillation for Device-Directed Speech Detection

TL;DR

This paper introduces an adaptive knowledge distillation method that improves device-directed speech detection accuracy by transferring knowledge from a large pre-trained acoustic encoder, enhancing performance across different model architectures.

Contribution

The paper proposes a novel adaptive knowledge distillation technique using task-specific adapters on a frozen teacher encoder for improved DDSD performance.

Findings

Adaptive KD outperforms baseline models without distillation.

Significant EER improvements of +26% and +19% in keyword and keyword-free scenarios.

Method generalizes across transformer and conformer architectures.

Abstract

Device-directed speech detection (DDSD) is a binary classification task that separates the user's queries to a voice assistant (VA) from background speech or side conversations. This is important for achieving naturalistic user experience. To this end, we propose knowledge distillation (KD) to enhance DDSD accuracy while ensuring efficient deployment. Specifically, we introduce a novel adaptive KD method that transfers knowledge from general representations of an ASR large pre-trained acoustic encoder (teacher). We apply task-specific adapters, on top of the (frozen) teacher encoder, trained jointly with the student model on DDSD. We demonstrate that the proposed adaptive KD outperforms the student model without distillation in the keyword and keyword-free (follow-up) invocations, with an improvement of +26% and +19% in terms of Equal Error Rate, respectively. We also show that this approach generalizes across the transformer and conformer-based model architectures.