GLAP: General contrastive audio-text pretraining across domains and languages

TL;DR

GLAP is a multilingual, multi-domain audio-text pretraining model that significantly improves retrieval and classification tasks across languages and sound types, surpassing existing methods.

Contribution

Introduces GLAP, a novel multilingual and multi-domain contrastive pretraining framework for audio and text, extending CLAP capabilities.

Findings

Achieves competitive results on Clotho and AudioCaps benchmarks.

Surpasses existing methods in speech retrieval and classification.

Excels in zero-shot sound-event recognition and multilingual keyword spotting.

Abstract

Contrastive Language Audio Pretraining (CLAP) is a widely-used method to bridge the gap between audio and text domains. Current CLAP methods enable sound and music retrieval in English, ignoring multilingual spoken content. To address this, we introduce general language audio pretraining (GLAP), which expands CLAP with multilingual and multi-domain abilities. GLAP demonstrates its versatility by achieving competitive performance on standard audio-text retrieval benchmarks like Clotho and AudioCaps, while significantly surpassing existing methods in speech retrieval and classification tasks. Additionally, GLAP achieves strong results on widely used sound-event zero-shot benchmarks, while simultaneously outperforming previous methods on speech content benchmarks. Further keyword spotting evaluations across 50 languages emphasize GLAP's advanced multilingual capabilities. Finally, multilingual sound and music understanding is evaluated across four languages. Checkpoints and Source: https://github.com/xiaomi-research/dasheng-glap.