MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting

TL;DR

MATE introduces nested audio-text embeddings with multiple granularities for open-vocabulary keyword spotting, achieving state-of-the-art results without additional inference costs.

Contribution

It proposes a novel matryoshka-style embedding framework with PCA-guided prefix alignment for improved keyword spotting.

Findings

Achieves state-of-the-art results on WSJ and LibriPhrase datasets.

Introduces nested embeddings capturing multiple granularities.

No additional inference overhead required.

Abstract

Open-vocabulary keyword spotting (KWS) with text-based enrollment has emerged as a flexible alternative to fixed-phrase triggers. Prior utterance-level matching methods, from an embedding-learning standpoint, learn embeddings at a single fixed dimensionality. We depart from this design and propose Matryoshka Audio-Text Embeddings (MATE), a dual-encoder framework that encodes multiple embedding granularities within a single vector via nested sub-embeddings ("prefixes"). Specifically, we introduce a PCA-guided prefix alignment: PCA-compressed versions of the full text embedding for each prefix size serve as teacher targets to align both audio and text prefixes. This alignment concentrates salient keyword cues in lower-dimensional prefixes, while higher dimensions add detail. MATE is trained with standard deep metric learning objectives for audio-text KWS, and is loss-agnostic. To our knowledge, this is the first application of matryoshka-style embeddings to KWS, achieving state-of-the-art results on WSJ and LibriPhrase without any inference overhead.