Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing

TL;DR

This paper introduces G2, a novel method that constructs scene graphs from images and uses them to improve visual commonsense reasoning and explanation, outperforming existing approaches.

Contribution

The paper proposes a scene-graph-enhanced reasoning framework that constructs location-free scene graphs from images and integrates them into reasoning tasks, with automatic filtering strategies.

Findings

Effective scene graph construction from images.

Improved accuracy in visual commonsense answering.

Demonstrated robustness through extensive experiments.

Abstract

Visual Commonsense Reasoning, which is regarded as one challenging task to pursue advanced visual scene comprehension, has been used to diagnose the reasoning ability of AI systems. However, reliable reasoning requires a good grasp of the scene's details. Existing work fails to effectively exploit the real-world object relationship information present within the scene, and instead overly relies on knowledge from training memory. Based on these observations, we propose a novel scene-graph-enhanced visual commonsense reasoning generation method named \textit{\textbf{G2}}, which first utilizes the image patches and LLMs to construct a location-free scene graph, and then answer and explain based on the scene graph's information. We also propose automatic scene graph filtering and selection strategies to absorb valuable scene graph information during training. Extensive experiments are conducted on the tasks and datasets of scene graph constructing and visual commonsense answering and explaining, respectively. Experimental results and ablation analysis demonstrate the effectiveness of our proposed framework.