CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

TL;DR

CAVL introduces a contrastive learning approach with adaptive fine-tuning networks to improve vision-language representations, achieving superior results across multiple tasks while significantly reducing computational costs.

Contribution

The paper proposes a novel contrastive pre-training method combined with lightweight adaptation networks for efficient vision-language learning.

Findings

Achieves state-of-the-art performance on six downstream tasks.

Reduces fine-tuning time by up to 76.17%.

Demonstrates the effectiveness of contrastive pre-training and adaptive fine-tuning.

Abstract

Visual and linguistic pre-training aims to learn vision and language representations together, which can be transferred to visual-linguistic downstream tasks. However, there exists semantic confusion between language and vision during the pre-training stage. Moreover, current pre-trained models tend to take lots of computation resources for fine-tuning when transferred to downstream tasks. In this work, we present a simple but effective approach for learning Contrastive and Adaptive representations of Vision and Language, namely CAVL. Specifically, we introduce a pair-wise contrastive loss to learn alignments between the whole sentence and each image in the same batch during the pre-training process. At the fine-tuning stage, we introduce two lightweight adaptation networks to reduce model parameters and increase training speed for saving computation resources. We evaluate our CAVL on six main downstream tasks, including Visual Question Answering (VQA), Visual Commonsense Reasoning (VCR), Natural Language for Visual Reasoning (NLVR), Region-to-Phrase Grounding (RPG), Text-to-Image Retrieval (TIR), and Zero-shot Text-to-Image Retrieval (ZS-TIR). Compared to baselines, we achieve superior performance and reduce the fine-tuning time by a large margin (in particular, 76.17%). Extensive experiments and ablation studies demonstrate the efficiency of contrastive pre-training and adaptive fine-tuning proposed in our CAVL.