Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases

TL;DR

This paper presents an unsupervised method for aligning audio clips with textual captions, enabling the learning of correspondences between individual sound events and phrases without annotations, demonstrated through retrieval and sound event detection tasks.

Contribution

The authors introduce a novel unsupervised approach that aligns audio and text by scoring similarities at the frame and word level, learning local and global correspondences without supervision.

Findings

Effective cross-modal retrieval performance

Successful local sound event-phrase correspondence learning

Competitive results in unsupervised sound event detection

Abstract

We investigate unsupervised learning of correspondences between sound events and textual phrases through aligning audio clips with textual captions describing the content of a whole audio clip. We align originally unaligned and unannotated audio clips and their captions by scoring the similarities between audio frames and words, as encoded by modality-specific encoders and using a ranking-loss criterion to optimize the model. After training, we obtain clip-caption similarity by averaging frame-word similarities and estimate event-phrase correspondences by calculating frame-phrase similarities. We evaluate the method with two cross-modal tasks: audio-caption retrieval, and phrase-based sound event detection (SED). Experimental results show that the proposed method can globally associate audio clips with captions as well as locally learn correspondences between individual sound events and textual phrases in an unsupervised manner.