Understanding the Role of Scene Graphs in Visual Question Answering

TL;DR

This paper investigates how scene graphs can enhance Visual Question Answering by evaluating various generation techniques, training strategies, and fusion architectures on the challenging GQA dataset, pioneering comprehensive analysis in this area.

Contribution

It introduces the first extensive study on integrating scene graphs into VQA, evaluating multiple techniques and proposing a training curriculum and fusion architectures.

Findings

Scene graphs improve VQA performance on complex questions.

Auto-generated scene graphs can be effectively used alongside human annotations.

Late fusion architectures enhance the integration of multiple image representations.

Abstract

Visual Question Answering (VQA) is of tremendous interest to the research community with important applications such as aiding visually impaired users and image-based search. In this work, we explore the use of scene graphs for solving the VQA task. We conduct experiments on the GQA dataset which presents a challenging set of questions requiring counting, compositionality and advanced reasoning capability, and provides scene graphs for a large number of images. We adopt image + question architectures for use with scene graphs, evaluate various scene graph generation techniques for unseen images, propose a training curriculum to leverage human-annotated and auto-generated scene graphs, and build late fusion architectures to learn from multiple image representations. We present a multi-faceted study into the use of scene graphs for VQA, making this work the first of its kind.