VCD: Visual Causality Discovery for Cross-Modal Question Reasoning

TL;DR

This paper introduces a novel framework for visual question reasoning that explicitly discovers causal structures and mitigates spurious correlations by combining causal intervention with multi-modal transformers, improving robustness.

Contribution

The paper proposes the VCD architecture and CMQR framework, integrating causal discovery and intervention with a multi-modal transformer for better causal reasoning in visual questions.

Findings

Outperforms existing methods on four datasets

Effectively discovers visual causal structures

Enhances robustness in question reasoning

Abstract

Existing visual question reasoning methods usually fail to explicitly discover the inherent causal mechanism and ignore jointly modeling cross-modal event temporality and causality. In this paper, we propose a visual question reasoning framework named Cross-Modal Question Reasoning (CMQR), to discover temporal causal structure and mitigate visual spurious correlation by causal intervention. To explicitly discover visual causal structure, the Visual Causality Discovery (VCD) architecture is proposed to find question-critical scene temporally and disentangle the visual spurious correlations by attention-based front-door causal intervention module named Local-Global Causal Attention Module (LGCAM). To align the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build an Interactive Visual-Linguistic Transformer (IVLT) that builds the multi-modal co-occurrence interactions between visual and linguistic content. Extensive experiments on four datasets demonstrate the superiority of CMQR for discovering visual causal structures and achieving robust question reasoning.