Cross-Modal Contrastive Learning for Robust Reasoning in VQA

TL;DR

This paper introduces a cross-modal contrastive learning approach that enhances robustness in visual question answering by reducing shortcut reasoning and leveraging fine-grained language-image correspondences.

Contribution

It proposes a novel contrastive learning strategy that avoids complex negative sampling and uses graph-based image relationships to improve VQA reasoning robustness.

Findings

Outperforms state-of-the-art on multiple VQA datasets.

Reduces reliance on shortcut reasoning in VQA models.

Demonstrates effectiveness of graph-based negative sampling.

Abstract

Multi-modal reasoning in visual question answering (VQA) has witnessed rapid progress recently. However, most reasoning models heavily rely on shortcuts learned from training data, which prevents their usage in challenging real-world scenarios. In this paper, we propose a simple but effective cross-modal contrastive learning strategy to get rid of the shortcut reasoning caused by imbalanced annotations and improve the overall performance. Different from existing contrastive learning with complex negative categories on coarse (Image, Question, Answer) triplet level, we leverage the correspondences between the language and image modalities to perform finer-grained cross-modal contrastive learning. We treat each Question-Answer (QA) pair as a whole, and differentiate between images that conform with it and those against it. To alleviate the issue of sampling bias, we further build connected graphs among images. For each positive pair, we regard the images from different graphs as negative samples and deduct the version of multi-positive contrastive learning. To our best knowledge, it is the first paper that reveals a general contrastive learning strategy without delicate hand-craft rules can contribute to robust VQA reasoning. Experiments on several mainstream VQA datasets demonstrate our superiority compared to the state of the arts. Code is available at \url{https://github.com/qizhust/cmcl_vqa_pl}.