Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"

TL;DR

This paper introduces a framework that isolates and evaluates the reasoning component in visual question answering, separating it from perception, and proposes a calibration technique to handle imperfect perception.

Contribution

It presents a novel differentiable logic formalism for VQA that disentangles reasoning from perception and enables detailed analysis of models' reasoning capabilities.

Findings

Disentangled reasoning evaluation reveals insights into model performance.

Calibration technique improves reasoning accuracy with imperfect perception.

Framework facilitates comparative analysis of VQA models.

Abstract

Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a differentiable first-order logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.