Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

TL;DR

This paper introduces a contrastive learning method called MMBS that leverages biased samples to improve the robustness of VQA models against out-of-distribution data without sacrificing in-distribution performance.

Contribution

The paper proposes a novel contrastive learning approach that exploits biased samples for unbiased reasoning, enhancing VQA model robustness against OOD data while maintaining ID accuracy.

Findings

Achieves competitive OOD performance on VQA-CP v2

Maintains strong ID performance on VQA v2

Compatible with various VQA backbones

Abstract

Models for Visual Question Answering (VQA) often rely on the spurious correlations, i.e., the language priors, that appear in the biased samples of training set, which make them brittle against the out-of-distribution (OOD) test data. Recent methods have achieved promising progress in overcoming this problem by reducing the impact of biased samples on model training. However, these models reveal a trade-off that the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data (which is dominated by the biased samples). Therefore, we propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples and explore several strategies to use the constructed positive samples for training. Instead of undermining the importance of biased samples in model training, our approach precisely exploits the biased samples for unbiased information that contributes to reasoning. The proposed method is compatible with various VQA backbones. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.