Training Vision-Language Transformers from Captions

TL;DR

This paper introduces VLC, a vision-language transformer trained solely on image captions without relying on traditional supervised visual pretraining, outperforming existing models like ViLT on benchmarks.

Contribution

Proposes VLC, a novel caption-based training approach for vision-language transformers that eliminates the need for ImageNet pretraining and bounding box supervision.

Findings

VLC outperforms ViLT on standard benchmarks.

VLC offers more interpretable patch visualizations.

VLC is competitive with larger ROI-based models.

Abstract

Vision-Language Transformers can be learned without low-level human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes or patches, assumes that the visual backbone must first be trained on ImageNet class prediction before being integrated into a multimodal linguistic pipeline. We show that this is not necessary and introduce a new model Vision-Language from Captions (VLC) built on top of Masked Auto-Encoders that does not require this supervision. In fact, in a head-to-head comparison between ViLT, the current state-of-the-art patch-based vision-language transformer which is pretrained with supervised object classification, and our model, VLC, we find that our approach 1. outperforms ViLT on standard benchmarks, 2. provides more interpretable and intuitive patch visualizations, and 3. is competitive with many larger models that utilize ROIs trained on annotated bounding-boxes.