LocTex: Learning Data-Efficient Visual Representations from Localized Textual Supervision

TL;DR

LocTex leverages low-cost localized textual annotations like captions and mouse traces to pre-train visual models, reducing annotation effort while maintaining or improving performance on various vision tasks.

Contribution

The paper introduces LocTex, a contrastive pre-training framework that uses captions and mouse traces for effective, data-efficient visual representation learning.

Findings

Reduces pre-training dataset size by 10x without performance loss

Achieves comparable or better results than ImageNet pre-training on COCO

Outperforms previous vision+language pre-training by 4% on PASCAL VOC

Abstract

Computer vision tasks such as object detection and semantic/instance segmentation rely on the painstaking annotation of large training datasets. In this paper, we propose LocTex that takes advantage of the low-cost localized textual annotations (i.e., captions and synchronized mouse-over gestures) to reduce the annotation effort. We introduce a contrastive pre-training framework between images and captions and propose to supervise the cross-modal attention map with rendered mouse traces to provide coarse localization signals. Our learned visual features capture rich semantics (from free-form captions) and accurate localization (from mouse traces), which are very effective when transferred to various downstream vision tasks. Compared with ImageNet supervised pre-training, LocTex can reduce the size of the pre-training dataset by 10x or the target dataset by 2x while achieving comparable or even improved performance on COCO instance segmentation. When provided with the same amount of annotations, LocTex achieves around 4% higher accuracy than the previous state-of-the-art "vision+language" pre-training approach on the task of PASCAL VOC image classification.