Dense Contrastive Visual-Linguistic Pretraining

TL;DR

This paper introduces DCVLP, a self-supervised multimodal pretraining method that uses dense contrastive learning to improve visual-linguistic representations without relying on noisy annotations.

Contribution

It proposes a novel dense contrastive learning framework for multimodal pretraining that eliminates the need for semantic annotations and improves representation quality.

Findings

Outperforms prior methods in multimodal tasks

Effectively handles noisy labels and sparse annotations

Enhances cross-modal dense region understanding

Abstract

Inspired by the success of BERT, several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. In particular, LXMERT and UNITER adopt visual region feature regression and label classification as pretext tasks. However, they tend to suffer from the problems of noisy labels and sparse semantic annotations, based on the visual features having been pretrained on a crowdsourced dataset with limited and inconsistent semantic labeling. To overcome these issues, we propose unbiased Dense Contrastive Visual-Linguistic Pretraining (DCVLP), which replaces the region regression and classification with cross-modality region contrastive learning that requires no annotations. Two data augmentation strategies (Mask Perturbation and Intra-/Inter-Adversarial Perturbation) are developed to improve the quality of negative samples used in contrastive learning. Overall, DCVLP allows cross-modality dense region contrastive learning in a self-supervised setting independent of any object annotations. We compare our method against prior visual-linguistic pretraining frameworks to validate the superiority of dense contrastive learning on multimodal representation learning.