Data leakage in cross-modal retrieval training: A case study

TL;DR

This paper investigates data leakage issues in the SoundDesc cross-modal audio retrieval dataset, revealing duplicates that inflate performance metrics, and proposes new splits to provide a more accurate and challenging benchmark.

Contribution

The study identifies data leakage in SoundDesc and introduces revised dataset splits to improve evaluation integrity in cross-modal retrieval.

Findings

Original splits contained duplicates causing data leakage.

New splits reduce leakage and increase task difficulty.

Revised dataset splits lead to more realistic performance assessments.

Abstract

The recent progress in text-based audio retrieval was largely propelled by the release of suitable datasets. Since the manual creation of such datasets is a laborious task, obtaining data from online resources can be a cheap solution to create large-scale datasets. We study the recently proposed SoundDesc benchmark dataset, which was automatically sourced from the BBC Sound Effects web page. In our analysis, we find that SoundDesc contains several duplicates that cause leakage of training data to the evaluation data. This data leakage ultimately leads to overly optimistic retrieval performance estimates in previous benchmarks. We propose new training, validation, and testing splits for the dataset that we make available online. To avoid weak contamination of the test data, we pool audio files that share similar recording setups. In our experiments, we find that the new splits serve as a more challenging benchmark.