Syntax Tree Constrained Graph Network for Visual Question Answering

TL;DR

This paper introduces a novel Syntax Tree Constrained Graph Network (STCGN) that leverages syntax trees of questions to enhance visual question answering accuracy by integrating syntactic information into the reasoning process.

Contribution

The paper proposes a new model that extracts syntax trees from questions and incorporates hierarchical syntax features into VQA, improving understanding and reasoning.

Findings

Outperforms existing VQA models on VQA2.0 dataset

Effectively captures syntactic structures to refine visual question understanding

Demonstrates the importance of syntax information in VQA tasks

Abstract

Visual Question Answering (VQA) aims to automatically answer natural language questions related to given image content. Existing VQA methods integrate vision modeling and language understanding to explore the deep semantics of the question. However, these methods ignore the significant syntax information of the question, which plays a vital role in understanding the essential semantics of the question and guiding the visual feature refinement. To fill the gap, we suggested a novel Syntax Tree Constrained Graph Network (STCGN) for VQA based on entity message passing and syntax tree. This model is able to extract a syntax tree from questions and obtain more precise syntax information. Specifically, we parse questions and obtain the question syntax tree using the Stanford syntax parsing tool. From the word level and phrase level, syntactic phrase features and question features are extracted using a hierarchical tree convolutional network. We then design a message-passing mechanism for phrase-aware visual entities and capture entity features according to a given visual context. Extensive experiments on VQA2.0 datasets demonstrate the superiority of our proposed model.