TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Training

TL;DR

TADS is a novel task-aware data selection framework for multi-task multimodal pre-training that improves data efficiency and model performance by intelligently selecting high-quality, relevant, and diverse data subsets.

Contribution

It introduces a comprehensive, learnable data selection method that integrates quality, relevance, and diversity, optimized via meta-learning for multi-task multimodal models.

Findings

TADS reduces data usage to 36% while outperforming baselines.

Achieves superior zero-shot performance on multiple benchmarks.

Enhances data efficiency and model generalization.

Abstract

Large-scale multimodal pre-trained models like CLIP rely heavily on high-quality training data, yet raw web-crawled datasets are often noisy, misaligned, and redundant, leading to inefficient training and suboptimal generalization. Existing data selection methods are either heuristic-based, suffering from bias and limited diversity, or data-driven but task-agnostic, failing to optimize for multi-task scenarios. To address these gaps, we introduce TADS (Task-Aware Data Selection), a novel framework for multi-task multimodal pre-training that integrates Intrinsic Quality, Task Relevance, and Distributional Diversity into a learnable value function. TADS employs a comprehensive quality assessment system with unimodal and cross-modal operators, quantifies task relevance via interpretable similarity vectors, and optimizes diversity through cluster-based weighting. A feedback-driven meta-learning mechanism adaptively refines the selection strategy based on proxy model performance across multiple downstream tasks. Experiments on CC12M demonstrate that TADS achieves superior zero-shot performance on benchmarks like ImageNet, CIFAR-100, MS-COCO, and Flickr30K, using only 36% of the data while outperforming baselines by an average of 1.0%. This highlights that TADS significantly enhances data efficiency by curating a high-utility subset that yields a much higher performance ceiling within the same computational constraints.