A CLIP-Powered Framework for Robust and Generalizable Data Selection

TL;DR

This paper introduces a CLIP-powered multimodal data selection framework that improves the robustness and efficiency of training deep learning models by effectively removing noisy data and selecting representative samples.

Contribution

The paper presents a novel multimodal data selection framework leveraging CLIP's knowledge, enhancing robustness and generalization over existing single-modality methods.

Findings

Outperforms state-of-the-art data selection methods on benchmark datasets.

Effectively removes noisy or damaged samples, improving data quality.

Achieves higher performance with less training data.

Abstract

Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets invariably incurs substantial storage and computational overhead. Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance. Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs. Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples. To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. Specifically, our framework consists of three key modules-dataset adaptation, sample scoring, and selection optimization-that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization. Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality.