Visual Question Answering based on Local-Scene-Aware Referring Expression Generation

TL;DR

This paper introduces a novel approach for visual question answering by generating rich textual expressions for images, integrating them with visual features and question embeddings through a multi-head attention network, leading to improved answer accuracy.

Contribution

It proposes using text expressions for images to enhance scene understanding in VQA, combined with a joint-embedding multi-head attention network for better multimodal integration.

Findings

Outperforms state-of-the-art methods on VQA v2 dataset.

Generated expressions are effective for scene description.

Method improves answer prediction accuracy.

Abstract

Visual question answering requires a deep understanding of both images and natural language. However, most methods mainly focus on visual concept; such as the relationships between various objects. The limited use of object categories combined with their relationships or simple question embedding is insufficient for representing complex scenes and explaining decisions. To address this limitation, we propose the use of text expressions generated for images, because such expressions have few structural constraints and can provide richer descriptions of images. The generated expressions can be incorporated with visual features and question embedding to obtain the question-relevant answer. A joint-embedding multi-head attention network is also proposed to model three different information modalities with co-attention. We quantitatively and qualitatively evaluated the proposed method on the VQA v2 dataset and compared it with state-of-the-art methods in terms of answer prediction. The quality of the generated expressions was also evaluated on the RefCOCO, RefCOCO+, and RefCOCOg datasets. Experimental results demonstrate the effectiveness of the proposed method and reveal that it outperformed all of the competing methods in terms of both quantitative and qualitative results.