SA-VQA: Structured Alignment of Visual and Semantic Representations for Visual Question Answering

TL;DR

This paper introduces a structured alignment approach using graph representations to improve visual question answering by capturing deep cross-modal connections, leading to better reasoning and interpretability.

Contribution

It proposes a novel graph-based structured alignment method for VQA that enhances reasoning and interpretability without pretraining.

Findings

Outperforms state-of-the-art on GQA dataset.

Beats non-pretrained SOTA on VQA-v2.

Improves reasoning performance and interpretability.

Abstract

Visual Question Answering (VQA) attracts much attention from both industry and academia. As a multi-modality task, it is challenging since it requires not only visual and textual understanding, but also the ability to align cross-modality representations. Previous approaches extensively employ entity-level alignments, such as the correlations between the visual regions and their semantic labels, or the interactions across question words and object features. These attempts aim to improve the cross-modality representations, while ignoring their internal relations. Instead, we propose to apply structured alignments, which work with graph representation of visual and textual content, aiming to capture the deep connections between the visual and textual modalities. Nevertheless, it is nontrivial to represent and integrate graphs for structured alignments. In this work, we attempt to solve this issue by first converting different modality entities into sequential nodes and the adjacency graph, then incorporating them for structured alignments. As demonstrated in our experimental results, such a structured alignment improves reasoning performance. In addition, our model also exhibits better interpretability for each generated answer. The proposed model, without any pretraining, outperforms the state-of-the-art methods on GQA dataset, and beats the non-pretrained state-of-the-art methods on VQA-v2 dataset.