Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

TL;DR

This paper introduces a two-stage curriculum learning approach for audio-visual retrieval that progressively trains with semi-hard and hard triplets, significantly improving retrieval performance.

Contribution

It proposes a novel two-stage training paradigm with curriculum augmentation to better distinguish semi-hard and hard triplets in cross-modal retrieval models.

Findings

Achieved approximately 9.8% improvement in MAP over state-of-the-art methods.

Effectively identifies and utilizes hard negatives through embedding augmentation.

Demonstrated significant performance gains on AVE dataset.

Abstract

The cross-modal retrieval model leverages the potential of triple loss optimization to learn robust embedding spaces. However, existing methods often train these models in a singular pass, overlooking the distinction between semi-hard and hard triples in the optimization process. The oversight of not distinguishing between semi-hard and hard triples leads to suboptimal model performance. In this paper, we introduce a novel approach rooted in curriculum learning to address this problem. We propose a two-stage training paradigm that guides the model's learning process from semi-hard to hard triplets. In the first stage, the model is trained with a set of semi-hard triplets, starting from a low-loss base. Subsequently, in the second stage, we augment the embeddings using an interpolation technique. This process identifies potential hard negatives, alleviating issues arising from high-loss functions due to a scarcity of hard triples. Our approach then applies hard triplet mining in the augmented embedding space to further optimize the model. Extensive experimental results conducted on two audio-visual datasets show a significant improvement of approximately 9.8% in terms of average Mean Average Precision (MAP) over the current state-of-the-art method, MSNSCA, for the Audio-Visual Cross-Modal Retrieval (AV-CMR) task on the AVE dataset, indicating the effectiveness of our proposed method.