Joint Multimodal Contrastive Learning for Robust Spoken Term Detection and Keyword Spotting

TL;DR

This paper introduces a joint multimodal contrastive learning framework that unifies acoustic and cross-modal supervision to improve spoken term detection and keyword spotting, outperforming existing methods.

Contribution

It presents the first comprehensive joint contrastive learning approach combining audio-text and audio-audio alignment for AWEs in speech retrieval tasks.

Findings

Outperforms existing AWE baselines on word discrimination tasks

Supports both STD and KWS with improved robustness

Unifies multimodal supervision in a shared embedding space

Abstract

Acoustic Word Embeddings (AWEs) improve the efficiency of speech retrieval tasks such as Spoken Term Detection (STD) and Keyword Spotting (KWS). However, existing approaches suffer from limitations, including unimodal supervision, disjoint optimization of audio-audio and audio-text alignment, and the need for task-specific models. To address these shortcomings, we propose a joint multimodal contrastive learning framework that unifies both acoustic and cross-modal supervision in a shared embedding space. Our approach simultaneously optimizes: (i) audio-text contrastive learning, inspired by the CLAP loss, to align audio and text representations and (ii) audio-audio contrastive learning, via Deep Word Discrimination (DWD) loss, to enhance intra-class compactness and inter-class separation. The proposed method outperforms existing AWE baselines on word discrimination task while flexibly supporting both STD and KWS. To our knowledge, this is the first comprehensive approach of its kind.