An Empirical Study on the Language Modal in Visual Question Answering

TL;DR

This empirical study investigates how language modality influences VQA model performance, revealing biases and proposing methods to improve out-of-distribution generalization, notably achieving significant gains without debiasing techniques.

Contribution

The paper provides new insights into language-related biases in VQA models and demonstrates effective strategies to enhance out-of-distribution performance without complex debiasing.

Findings

Postfix-related bias significantly affects VQA performance.

Training with word-sequence variants improves out-of-distribution accuracy.

LXMERT achieved a 10-point gain without debiasing methods.

Abstract

Generalization beyond in-domain experience to out-of-distribution data is of paramount significance in the AI domain. Of late, state-of-the-art Visual Question Answering (VQA) models have shown impressive performance on in-domain data, partially due to the language priors bias which, however, hinders the generalization ability in practice. This paper attempts to provide new insights into the influence of language modality on VQA performance from an empirical study perspective. To achieve this, we conducted a series of experiments on six models. The results of these experiments revealed that, 1) apart from prior bias caused by question types, there is a notable influence of postfix-related bias in inducing biases, and 2) training VQA models with word-sequence-related variant questions demonstrated improved performance on the out-of-distribution benchmark, and the LXMERT even achieved a 10-point gain without adopting any debiasing methods. We delved into the underlying reasons behind these experimental results and put forward some simple proposals to reduce the models' dependency on language priors. The experimental results demonstrated the effectiveness of our proposed method in improving performance on the out-of-distribution benchmark, VQA-CPv2. We hope this study can inspire novel insights for future research on designing bias-reduction approaches.