An experimental study of the vision-bottleneck in VQA

TL;DR

This paper investigates how the quality and quantity of visual information affect VQA performance, emphasizing the importance of tailored vision methods over generic object detection.

Contribution

It provides an in-depth analysis of the vision bottleneck in VQA, exploring different visual object extraction strategies and their impact on question answering accuracy.

Findings

Quality of visual objects significantly influences VQA accuracy.

Early incorporation of object information improves reasoning performance.

Tailoring vision methods to VQA enhances overall effectiveness.

Abstract

As in many tasks combining vision and language, both modalities play a crucial role in Visual Question Answering (VQA). To properly solve the task, a given model should both understand the content of the proposed image and the nature of the question. While the fusion between modalities, which is another obviously important part of the problem, has been highly studied, the vision part has received less attention in recent work. Current state-of-the-art methods for VQA mainly rely on off-the-shelf object detectors delivering a set of object bounding boxes and embeddings, which are then combined with question word embeddings through a reasoning module. In this paper, we propose an in-depth study of the vision-bottleneck in VQA, experimenting with both the quantity and quality of visual objects extracted from images. We also study the impact of two methods to incorporate the information about objects necessary for answering a question, in the reasoning module directly, and earlier in the object selection stage. This work highlights the importance of vision in the context of VQA, and the interest of tailoring vision methods used in VQA to the task at hand.