ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering

TL;DR

ReasonVQA introduces a large, structured knowledge-integrated dataset for VQA that challenges current models with complex, multi-hop questions, advancing benchmarking and research in visual reasoning.

Contribution

The paper presents a new scalable dataset, ReasonVQA, with integrated encyclopedic knowledge and complex questions, providing a challenging benchmark for VQA models.

Findings

State-of-the-art models perform poorly on ReasonVQA

ReasonVQA surpasses existing knowledge-based VQA datasets in size

The dataset enables evaluation of multi-hop reasoning capabilities

Abstract

In this paper, we propose a new dataset, ReasonVQA, for the Visual Question Answering (VQA) task. Our dataset is automatically integrated with structured encyclopedic knowledge and constructed using a low-cost framework, which is capable of generating complex, multi-hop questions. We evaluated state-of-the-art VQA models on ReasonVQA, and the empirical results demonstrate that ReasonVQA poses significant challenges to these models, highlighting its potential for benchmarking and advancing the field of VQA. Additionally, our dataset can be easily scaled with respect to input images; the current version surpasses the largest existing datasets requiring external knowledge by more than an order of magnitude.