CiT: Curation in Training for Effective Vision-Language Data

TL;DR

CiT introduces an efficient training method for vision-language models that automatically curates high-quality data from large pools, significantly speeding up training without extensive offline filtering.

Contribution

The paper proposes a novel data curation algorithm integrated into training, reducing data filtering costs and enabling faster, scalable vision-language model training.

Findings

Speeds up training by over an order of magnitude.

Effectively utilizes raw web data without offline filtering.

Maintains competitive performance with less data filtering effort.

Abstract

Large vision-language models are generally applicable to many downstream tasks, but come at an exorbitant training cost that only large institutions can afford. This paper trades generality for efficiency and presents Curation in Training (CiT), a simple and efficient vision-text learning algorithm that couples a data objective into training. CiT automatically yields quality data to speed-up contrastive image-text training and alleviates the need for an offline data filtering pipeline, allowing broad data sources (including raw image-text pairs from the web). CiT contains two loops: an outer loop curating the training data and an inner loop consuming the curated training data. The text encoder connects the two loops. Given metadata for tasks of interest, e.g., class names, and a large pool of image-text pairs, CiT alternatively selects relevant training data from the pool by measuring the similarity of their text embeddings and embeddings of the metadata. In our experiments, we observe that CiT can speed up training by over an order of magnitude, especially if the raw data size is large.