Domain-robust VQA with diverse datasets and methods but no target labels

TL;DR

This paper investigates the robustness of various VQA models to domain shifts across datasets, quantifies these shifts, and proposes a new domain adaptation method tailored for VQA's unique challenges.

Contribution

It introduces a comprehensive analysis of domain shifts in VQA, evaluates existing models' robustness, and develops a novel domain adaptation approach specific to VQA complexities.

Findings

VQA datasets exhibit significant domain shifts in visual and textual modalities.

Transformer-based VQA models show greater robustness to domain shifts.

The proposed domain adaptation method improves VQA performance across different datasets.

Abstract

The observation that computer vision methods overfit to dataset specifics has inspired diverse attempts to make object recognition models robust to domain shifts. However, similar work on domain-robust visual question answering methods is very limited. Domain adaptation for VQA differs from adaptation for object recognition due to additional complexity: VQA models handle multimodal inputs, methods contain multiple steps with diverse modules resulting in complex optimization, and answer spaces in different datasets are vastly different. To tackle these challenges, we first quantify domain shifts between popular VQA datasets, in both visual and textual space. To disentangle shifts between datasets arising from different modalities, we also construct synthetic shifts in the image and question domains separately. Second, we test the robustness of different families of VQA methods (classic two-stream, transformer, and neuro-symbolic methods) to these shifts. Third, we test the applicability of existing domain adaptation methods and devise a new one to bridge VQA domain gaps, adjusted to specific VQA models. To emulate the setting of real-world generalization, we focus on unsupervised domain adaptation and the open-ended classification task formulation.