Counterfactual VQA: A Cause-Effect Look at Language Bias

TL;DR

This paper introduces a counterfactual inference framework for VQA that effectively reduces language bias by disentangling causal effects, improving performance on biased and balanced datasets without data augmentation.

Contribution

It proposes a novel causal inference approach to mitigate language bias in VQA, applicable across various models and fusion strategies.

Findings

Improves VQA performance on bias-sensitive datasets

Reduces reliance on language priors

Maintains robustness on balanced datasets

Abstract

VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at https://github.com/yuleiniu/cfvqa.