Triage knowledge distillation for speaker verification

TL;DR

This paper introduces Triage Knowledge Distillation (TRKD), a novel method for speaker verification that improves model performance by selectively transferring relational information based on example difficulty, especially effective on resource-limited devices.

Contribution

TRKD operationalizes assess-prioritize-focus in knowledge distillation, using a cumulative-probability cutoff to focus on confusable classes and progressively refine learning, outperforming existing KD methods.

Findings

TRKD achieves the lowest EER across all protocols on VoxCeleb1.

TRKD outperforms recent KD variants in speaker verification tasks.

TRKD effectively handles large-class settings with long-tail distributions.

Abstract

Deploying speaker verification on resource-constrained devices remains challenging due to the computational cost of high-capacity models; knowledge distillation (KD) offers a remedy. Classical KD entangles target confidence with non-target structure in a Kullback-Leibler term, limiting the transfer of relational information. Decoupled KD separates these signals into target and non-target terms, yet treats non-targets uniformly and remains vulnerable to the long tail of low-probability classes in large-class settings. We introduce Triage KD (TRKD), a distillation scheme that operationalizes assess-prioritize-focus. TRKD introduces a cumulative-probability cutoff $\tau$ to assess per-example difficulty and partition the teacher posterior into three groups: the target class, a high-probability non-target confusion-set, and a background-set. To prioritize informative signals, TRKD distills the confusion-set conditional distribution and discards the background. Concurrently, it transfers a three-mass (target/confusion/background) that capture sample difficulty and inter-class confusion. Finally, TRKD focuses learning via a curriculum on $\tau$: training begins with a larger $\tau$ to convey broad non-target context, then $\tau$ is progressively decreased to shrink the confusion-set, concentrating supervision on the most confusable classes. In extensive experiments on VoxCeleb1 with both homogeneous and heterogeneous teacher-student pairs, TRKD was consistently superior to recent KD variants and attained the lowest EER across all protocols.