Zero-Shot Transfer VQA Dataset

TL;DR

This paper introduces ZST-VQA, a new dataset for zero-shot transfer in visual question answering, highlighting the challenge of transferring knowledge between question understanding and answering, and evaluating current models' limitations.

Contribution

The paper presents the ZST-VQA dataset, reorganized from VQA v1.0, to specifically evaluate zero-shot transfer capabilities in VQA models, a novel benchmark for this problem.

Findings

Existing models perform poorly on zero-shot transfer tasks.

Performance drops indicate current methods do not effectively handle zero-shot transfer.

Implicit bias during training may hinder transfer learning in VQA models.

Abstract

Acquiring a large vocabulary is an important aspect of human intelligence. Onecommon approach for human to populating vocabulary is to learn words duringreading or listening, and then use them in writing or speaking. This ability totransfer from input to output is natural for human, but it is difficult for machines.Human spontaneously performs this knowledge transfer in complicated multimodaltasks, such as Visual Question Answering (VQA). In order to approach human-levelArtificial Intelligence, we hope to equip machines with such ability. Therefore, toaccelerate this research, we propose a newzero-shot transfer VQA(ZST-VQA)dataset by reorganizing the existing VQA v1.0 dataset in the way that duringtraining, some words appear only in one module (i.e. questions) but not in theother (i.e. answers). In this setting, an intelligent model should understand andlearn the concepts from one module (i.e. questions), and at test time, transfer themto the other (i.e. predict the concepts as answers). We conduct evaluation on thisnew dataset using three existing state-of-the-art VQA neural models. Experimentalresults show a significant drop in performance on this dataset, indicating existingmethods do not address the zero-shot transfer problem. Besides, our analysis findsthat this may be caused by the implicit bias learned during training.