Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting

TL;DR

This paper introduces a novel adversarial deep metric learning framework for cross-modal audio-text alignment in open-vocabulary keyword spotting, reducing modality heterogeneity and improving phoneme-level alignment.

Contribution

It proposes Modality Adversarial Learning (MAL) to generate modality-invariant embeddings and applies deep metric learning for effective audio-text alignment.

Findings

Improved keyword spotting accuracy on WSJ and LibriPhrase datasets.

Effective reduction of modality gap through adversarial training.

Enhanced phoneme-level alignment between audio and text.

Abstract

For text enrollment-based open-vocabulary keyword spotting (KWS), acoustic and text embeddings are typically compared at either the phoneme or utterance level. To facilitate this, we optimize acoustic and text encoders using deep metric learning (DML), enabling direct comparison of multi-modal embeddings in a shared embedding space. However, the inherent heterogeneity between audio and text modalities presents a significant challenge. To address this, we propose Modality Adversarial Learning (MAL), which reduces the domain gap in heterogeneous modality representations. Specifically, we train a modality classifier adversarially to encourage both encoders to generate modality-invariant embeddings. Additionally, we apply DML to achieve phoneme-level alignment between audio and text, and conduct extensive comparisons across various DML objectives. Experiments on the Wall Street Journal (WSJ) and LibriPhrase datasets demonstrate the effectiveness of the proposed approach.