Caption supervision enables robust learners

TL;DR

This paper demonstrates that caption-supervised CNNs trained with standard cross-entropy can outperform vision-language models like CLIP in distributional robustness, and introduces CaptionNet, a new dataset for future research.

Contribution

It shows caption-supervised CNNs can be more robust than VL models and provides a new dataset, CaptionNet, for advancing caption supervision research.

Findings

Caption-supervised CNNs can outperform VL models in robustness.

Choice of loss function and supervision strategy impacts robustness.

Introduction of CaptionNet dataset with 50,000+ human-labeled samples.

Abstract

Vision language (VL) models like CLIP are robust to natural distribution shifts, in part because CLIP learns on unstructured data using a technique called caption supervision; the model inteprets image-linked texts as ground-truth labels. In a carefully controlled comparison study, we show that caption-supervised CNNs trained on a standard cross-entropy loss (with image labels assigned by scanning captions for class names) can exhibit greater distributional robustness than VL models trained on the same data. To facilitate future experiments with high-accuracy caption-supervised models, we introduce CaptionNet (https://github.com/penfever/CaptionNet/), which includes a class-balanced, fully supervised dataset with over 50,000 new human-labeled ImageNet-compliant samples which includes web-scraped captions. In a series of experiments on CaptionNet, we show how the choice of loss function, data filtration and supervision strategy enable robust computer vision. We also provide the codebase necessary to reproduce our experiments at VL Hub (https://github.com/penfever/vlhub/).