On the Significance of Question Encoder Sequence Model in the Out-of-Distribution Performance in Visual Question Answering

TL;DR

This paper investigates how the choice of question encoder architecture affects the out-of-distribution generalization of Visual Question Answering models, highlighting the importance of sequence model design.

Contribution

It provides a detailed analysis of RNN and Transformer question encoders and introduces a novel GAT-based encoder that enhances model generalizability.

Findings

Sequence model choice significantly impacts OOD performance.

GAT-based encoder outperforms RNN and Transformer encoders.

Improved generalization achieved without complex bias mitigation.

Abstract

Generalizing beyond the experiences has a significant role in developing practical AI systems. It has been shown that current Visual Question Answering (VQA) models are over-dependent on the language-priors (spurious correlations between question-types and their most frequent answers) from the train set and pose poor performance on Out-of-Distribution (OOD) test sets. This conduct limits their generalizability and restricts them from being utilized in real-world situations. This paper shows that the sequence model architecture used in the question-encoder has a significant role in the generalizability of VQA models. To demonstrate this, we performed a detailed analysis of various existing RNN-based and Transformer-based question-encoders, and along, we proposed a novel Graph attention network (GAT)-based question-encoder. Our study found that a better choice of sequence model in the question-encoder improves the generalizability of VQA models even without using any additional relatively complex bias-mitigation approaches.