Modulated Self-attention Convolutional Network for VQA

TL;DR

This paper introduces a modulated self-attention convolutional network that integrates linguistic input to enhance visual feature extraction for VQA, showing promising initial improvements.

Contribution

It proposes a novel CNN architecture augmented with self-attention modulated by language input for improved visual reasoning in VQA.

Findings

Encouraging relative improvements in VQA performance.

Demonstrates the potential of language-modulated self-attention in CNNs.

Provides a foundation for future research in end-to-end visual feature extraction.

Abstract

As new data-sets for real-world visual reasoning and compositional question answering are emerging, it might be needed to use the visual feature extraction as a end-to-end process during training. This small contribution aims to suggest new ideas to improve the visual processing of traditional convolutional network for visual question answering (VQA). In this paper, we propose to modulate by a linguistic input a CNN augmented with self-attention. We show encouraging relative improvements for future research in this direction.