Contrast and Classify: Training Robust VQA Models

TL;DR

This paper introduces ConClaT, a novel training method for VQA models that combines cross-entropy and contrastive losses to improve robustness to linguistic variations, leading to better consistency and accuracy.

Contribution

We propose a contrastive learning-based training paradigm for VQA models that enhances robustness to question paraphrases and improves performance on benchmark datasets.

Findings

Improves Consensus Score by 1.63% on VQA-Rephrasings

Increases VQA accuracy by 0.78% on VQA 2.0

Effective across different data-augmentation strategies

Abstract

Recent Visual Question Answering (VQA) models have shown impressive performance on the VQA benchmark but remain sensitive to small linguistic variations in input questions. Existing approaches address this by augmenting the dataset with question paraphrases from visual question generation models or adversarial perturbations. These approaches use the combined data to learn an answer classifier by minimizing the standard cross-entropy loss. To more effectively leverage augmented data, we build on the recent success in contrastive learning. We propose a novel training paradigm (ConClaT) that optimizes both cross-entropy and contrastive losses. The contrastive loss encourages representations to be robust to linguistic variations in questions while the cross-entropy loss preserves the discriminative power of representations for answer prediction. We find that optimizing both losses -- either alternately or jointly -- is key to effective training. On the VQA-Rephrasings benchmark, which measures the VQA model's answer consistency across human paraphrases of a question, ConClaT improves Consensus Score by 1 .63% over an improved baseline. In addition, on the standard VQA 2.0 benchmark, we improve the VQA accuracy by 0.78% overall. We also show that ConClaT is agnostic to the type of data-augmentation strategy used.