Classification Done Right for Vision-Language Pre-Training

TL;DR

SuperClass is a straightforward classification approach for vision-language pre-training that uses raw text tokens as labels, eliminating the need for a text encoder and large batch sizes, while achieving superior downstream task performance.

Contribution

It introduces SuperClass, a simple classification method that outperforms contrastive models like CLIP without requiring a text encoder or large batch sizes.

Findings

SuperClass achieves better results on vision-language tasks than CLIP.

Scaling SuperClass with model size and data improves performance.

SuperClass simplifies training while maintaining high accuracy.

Abstract

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass