Training CLIP models on Data from Scientific Papers

TL;DR

This paper investigates whether training CLIP models on high-quality scientific paper data improves their performance, finding moderate gains on small models and suggesting potential for larger models.

Contribution

It introduces a method for extracting scientific paper data for CLIP training and evaluates its impact on model performance.

Findings

Moderate performance increase on small CLIP models with scientific data

High-quality domain-specific data is promising for future large-scale CLIP training

Using scientific papers as training data is a worthwhile research direction

Abstract

Contrastive Language-Image Pretraining (CLIP) models are able to capture the semantic relationship of images and texts and have enabled a wide range of applications, from image retrieval to classification. These models are trained with datasets extracted from web crawls, which are of large quantity but limited quality. This paper explores whether limited amounts higher quality data in a specific domain improve the general performance of CLIP models. To this purpose, we extract text-image data from scientific papers hosted in the arXiv and PubMed Central repositories. Experiments on small-scale CLIP models (ViT B/32) show that model performance increases on average, but only moderately. This result indicates that using the data sources considered in the paper to train large-scale CLIP models is a worthwile research direction.