PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords

TL;DR

PhonMatchNet introduces a zero-shot keyword spotting model leveraging phoneme information, outperforming baselines and rivaling full-shot models across diverse pronunciation scenarios.

Contribution

It proposes a novel two-stream architecture with phoneme-level detection, enhancing zero-shot keyword spotting performance in various pronunciation environments.

Findings

Significant reduction in EER and AUC metrics across datasets.

Outperforms baseline models and rivals full-shot models.

Effective in recognizing proper nouns and indistinguishable pronunciations.

Abstract

This study presents a novel zero-shot user-defined keyword spotting model that utilizes the audio-phoneme relationship of the keyword to improve performance. Unlike the previous approach that estimates at utterance level, we use both utterance and phoneme level information. Our proposed method comprises a two-stream speech encoder architecture, self-attention-based pattern extractor, and phoneme-level detection loss for high performance in various pronunciation environments. Based on experimental results, our proposed model outperforms the baseline model and achieves competitive performance compared with full-shot keyword spotting models. Our proposed model significantly improves the EER and AUC across all datasets, including familiar words, proper nouns, and indistinguishable pronunciations, with an average relative improvement of 67% and 80%, respectively. The implementation code of our proposed model is available at https://github.com/ncsoft/PhonMatchNet.