From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

TL;DR

This paper introduces a Hierarchical Graph Neural Module Network that performs multi-layer, multi-modal reasoning for visual question answering, achieving state-of-the-art results with explicit, explainable reasoning processes.

Contribution

It extends modular reasoning to multi-layer, multi-modal graphs, enabling explicit, interpretable reasoning in VQA systems, which was not addressed by prior models.

Findings

Achieves state-of-the-art performance on CRIC dataset

Provides explicit reasoning trace via module weights and graph attention

Successfully reasons over visual, semantic, and commonsense graphs

Abstract

In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.