ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors

TL;DR

This paper introduces ATRI, a novel approach for multilingual audio-text retrieval that reduces data distribution errors to improve consistency and recall across multiple languages, achieving state-of-the-art results.

Contribution

The paper presents a theoretical analysis of inconsistencies in ML-ATR and proposes a new scheme using contrastive learning to mitigate data distribution errors.

Findings

Achieves state-of-the-art recall on multilingual datasets.

Improves consistency across eight languages.

Reduces data distribution errors in ML-ATR.

Abstract

Multilingual audio-text retrieval (ML-ATR) is a challenging task that aims to retrieve audio clips or multilingual texts from databases. However, existing ML-ATR schemes suffer from inconsistencies for instance similarity matching across languages. We theoretically analyze the inconsistency in terms of both multilingual modal alignment direction error and weight error, and propose the theoretical weight error upper bound for quantifying the inconsistency. Based on the analysis of the weight error upper bound, we find that the inconsistency problem stems from the data distribution error caused by random sampling of languages. We propose a consistent ML-ATR scheme using 1-to-k contrastive learning and audio-English co-anchor contrastive learning, aiming to mitigate the negative impact of data distribution error on recall and consistency in ML-ATR. Experimental results on the translated AudioCaps and Clotho datasets show that our scheme achieves state-of-the-art performance on recall and consistency metrics for eight mainstream languages, including English. Our code will be available at https://github.com/ATRI-ACL/ATRI-ACL.