On Negative Sampling for Contrastive Audio-Text Retrieval

TL;DR

This paper examines negative sampling strategies in contrastive audio-text retrieval, demonstrating that semi-hard negatives improve performance and highlighting issues like feature collapse with hard negatives.

Contribution

It introduces and evaluates various negative sampling strategies, especially semi-hard sampling with cross-modality scores, for contrastive audio-text retrieval.

Findings

Semi-hard negatives with cross-modality scores improve retrieval performance.

Performance varies significantly among different negative sampling strategies.

Hard negative sampling can cause feature collapse in the model.

Abstract

This paper investigates negative sampling for contrastive learning in the context of audio-text retrieval. The strategy for negative sampling refers to selecting negatives (either audio clips or textual descriptions) from a pool of candidates for a positive audio-text pair. We explore sampling strategies via model-estimated within-modality and cross-modality relevance scores for audio and text samples. With a constant training setting on the retrieval system from [1], we study eight sampling strategies, including hard and semi-hard negative sampling. Experimental results show that retrieval performance varies dramatically among different strategies. Particularly, by selecting semi-hard negatives with cross-modality scores, the retrieval system gains improved performance in both text-to-audio and audio-to-text retrieval. Besides, we show that feature collapse occurs while sampling hard negatives with cross-modality scores.