VirTex: Learning Visual Representations from Textual Annotations

TL;DR

VirTex introduces a caption-based pretraining method that learns high-quality visual representations efficiently from fewer images, outperforming traditional ImageNet-based methods across multiple vision tasks.

Contribution

The paper presents VirTex, a novel caption-based pretraining approach that achieves competitive or superior results with significantly less data compared to existing methods.

Findings

VirTex matches or exceeds ImageNet pretraining performance.

Uses up to ten times fewer images for training.

Effective across classification, detection, and segmentation tasks.

Abstract

The de-facto approach to many vision tasks is to start from pretrained visual representations, typically learned via supervised training on ImageNet. Recent methods have explored unsupervised pretraining to scale to vast quantities of unlabeled images. In contrast, we aim to learn high-quality visual representations from fewer images. To this end, we revisit supervised pretraining, and seek data-efficient alternatives to classification-based pretraining. We propose VirTex -- a pretraining approach using semantically dense captions to learn visual representations. We train convolutional networks from scratch on COCO Captions, and transfer them to downstream recognition tasks including image classification, object detection, and instance segmentation. On all tasks, VirTex yields features that match or exceed those learned on ImageNet -- supervised or unsupervised -- despite using up to ten times fewer images.