Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

TL;DR

This paper introduces VIP-ANT, a novel method that aligns audio and text representations without relying on parallel data by leveraging visual information as a bridge, achieving state-of-the-art zero-shot audio classification and caption retrieval.

Contribution

VIP-ANT is the first approach to connect audio and text without parallel data by using image modality as a pivot in a tri-modal embedding space.

Findings

Achieves state-of-the-art zero-shot performance on ESC50 and US8K datasets.

Surpasses supervised methods in Clotho caption retrieval with audio queries.

Minimal supervised data significantly boosts zero-shot accuracy.

Abstract

Machines that can represent and describe environmental soundscapes have practical potential, e.g., for audio tagging and captioning systems. Prevailing learning paradigms have been relying on parallel audio-text data, which is, however, scarcely available on the web. We propose VIP-ANT that induces \textbf{A}udio-\textbf{T}ext alignment without using any parallel audio-text data. Our key idea is to share the image modality between bi-modal image-text representations and bi-modal image-audio representations; the image modality functions as a pivot and connects audio and text in a tri-modal embedding space implicitly. In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2.2\% R@1. We further investigate cases of minimal audio-text supervision, finding that, e.g., just a few hundred supervised audio-text pairs increase the zero-shot audio classification accuracy by 8\% on US8K. However, to match human parity on some zero-shot tasks, our empirical scaling experiments suggest that we would need about $2^{21} \approx 2M$ supervised audio-caption pairs. Our work opens up new avenues for learning audio-text connections with little to no parallel audio-text data.