An Empirical Study on Leveraging Scene Graphs for Visual Question Answering

TL;DR

This paper explores using scene graphs and graph networks for Visual Question Answering, showing they can effectively capture image information and offer a more interpretable, potentially superior alternative to existing methods.

Contribution

It introduces an approach that employs scene graphs and graph networks for Visual QA, emphasizing interpretability and competitive performance.

Findings

Scene graphs effectively encode image information.

Graph networks can outperform state-of-the-art Visual QA models.

The approach offers a more interpretable reasoning process.

Abstract

Visual question answering (Visual QA) has attracted significant attention these years. While a variety of algorithms have been proposed, most of them are built upon different combinations of image and language features as well as multi-modal attention and fusion. In this paper, we investigate an alternative approach inspired by conventional QA systems that operate on knowledge graphs. Specifically, we investigate the use of scene graphs derived from images for Visual QA: an image is abstractly represented by a graph with nodes corresponding to object entities and edges to object relationships. We adapt the recently proposed graph network (GN) to encode the scene graph and perform structured reasoning according to the input question. Our empirical studies demonstrate that scene graphs can already capture essential information of images and graph networks have the potential to outperform state-of-the-art Visual QA algorithms but with a much cleaner architecture. By analyzing the features generated by GNs we can further interpret the reasoning process, suggesting a promising direction towards explainable Visual QA.