MT-HuBERT: Self-Supervised Mix-Training for Few-Shot Keyword Spotting in Mixed Speech

TL;DR

MT-HuBERT introduces a self-supervised pre-training framework for few-shot keyword spotting in mixed speech, effectively handling overlapping keywords and outperforming state-of-the-art methods in both mixed and clean conditions.

Contribution

This work presents MT-HuBERT, a novel self-supervised Mix-Training approach that leverages unlabeled data for improved mixed speech keyword spotting.

Findings

Outperforms state-of-the-art baselines in few-shot KWS tasks

Effective in both mixed and clean speech conditions

Demonstrates efficiency with unlabeled data in pre-training

Abstract

Few-shot keyword spotting aims to detect previously unseen keywords with very limited labeled samples. A pre-training and adaptation paradigm is typically adopted for this task. While effective in clean conditions, most existing approaches struggle with mixed keyword spotting--detecting multiple overlapping keywords within a single utterance--a capability essential for real-world applications. We have previously proposed a pre-training approach based on Mix-Training (MT) to tackle the mixed keyword detection problem and demonstrated its efficiency. However, this approach is fully supervised, unable to utilize vast unlabeled data. To this end, we propose Mix-Training HuBERT (MT-HuBERT), a self-supervised learning (SSL) pre-training framework that implements the MT criterion during pre-training. MT-HuBERT predicts, in a self-supervised manner, the clean acoustic units of each constituent signal from contextual cues, in contrast to predicting compositional patterns of mixed speech. Experiments conducted on the Google Speech Commands (GSC v2) corpus demonstrate that our proposed MT-HuBERT consistently outperforms several state-of-the-art baselines in few-shot KWS tasks under both mixed and clean conditions.