SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering

TL;DR

This paper introduces SelfGraphVQA, a self-supervised framework that enhances scene graph-based visual question answering by using augmentation and contrastive learning to improve generalization without relying on annotated scene graphs.

Contribution

The paper proposes a novel self-supervised approach for scene graph extraction and augmentation in VQA, reducing reliance on annotated data and improving reasoning capabilities.

Findings

SelfGraphVQA improves VQA accuracy using self-supervised graph learning.

Augmentation strategies enhance the robustness of scene graph representations.

Contrastive learning boosts the effectiveness of scene graph features in VQA.

Abstract

The intersection of vision and language is of major interest due to the increased focus on seamless integration between recognition and reasoning. Scene graphs (SGs) have emerged as a useful tool for multimodal image analysis, showing impressive performance in tasks such as Visual Question Answering (VQA). In this work, we demonstrate that despite the effectiveness of scene graphs in VQA tasks, current methods that utilize idealized annotated scene graphs struggle to generalize when using predicted scene graphs extracted from images. To address this issue, we introduce the SelfGraphVQA framework. Our approach extracts a scene graph from an input image using a pre-trained scene graph generator and employs semantically-preserving augmentation with self-supervised techniques. This method improves the utilization of graph representations in VQA tasks by circumventing the need for costly and potentially biased annotated data. By creating alternative views of the extracted graphs through image augmentations, we can learn joint embeddings by optimizing the informational content in their representations using an un-normalized contrastive approach. As we work with SGs, we experiment with three distinct maximization strategies: node-wise, graph-wise, and permutation-equivariant regularization. We empirically showcase the effectiveness of the extracted scene graph for VQA and demonstrate that these approaches enhance overall performance by highlighting the significance of visual information. This offers a more practical solution for VQA tasks that rely on SGs for complex reasoning questions.