Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning

TL;DR

This paper introduces DISSect, a differential-informed sample selection method that improves the efficiency of multimodal contrastive learning by effectively identifying and excluding noisy data, leading to faster training and better performance.

Contribution

The paper proposes a novel differential-based sample selection approach that addresses noisy correspondence in contrastive learning, with theoretical analysis and extensive experimental validation.

Findings

DISSect outperforms state-of-the-art methods on benchmark datasets.

The differential between current and historical model predictions effectively identifies noisy samples.

The method accelerates training while maintaining or improving model performance.

Abstract

The remarkable success of contrastive-learning-based multimodal models has been greatly driven by training on ever-larger datasets with expensive compute consumption. Sample selection as an alternative efficient paradigm plays an important direction to accelerate the training process. However, recent advances on sample selection either mostly rely on an oracle model to offline select a high-quality coreset, which is limited in the cold-start scenarios, or focus on online selection based on real-time model predictions, which has not sufficiently or efficiently considered the noisy correspondence. To address this dilemma, we propose a novel Differential-Informed Sample Selection (DISSect) method, which accurately and efficiently discriminates the noisy correspondence for training acceleration. Specifically, we rethink the impact of noisy correspondence on contrastive learning and propose that the differential between the predicted correlation of the current model and that of a historical model is more informative to characterize sample quality. Based on this, we construct a robust differential-based sample selection and analyze its theoretical insights. Extensive experiments on three benchmark datasets and various downstream tasks demonstrate the consistent superiority of DISSect over current state-of-the-art methods. Source code is available at: https://github.com/MediaBrain-SJTU/DISSect.