Barlow constrained optimization for Visual Question Answering

TL;DR

This paper introduces COB, a regularization method for VQA that reduces redundancy in the joint embedding space, improving accuracy and interpretability by disentangling semantic concepts.

Contribution

It proposes a novel constrained optimization regularization based on Barlow's theory to enhance the information content of the VQA joint space.

Findings

Improves VQA accuracy by 1.4% on VQA-CP v2

Enhances interpretability of the model

Reduces redundancy in the joint embedding space

Abstract

Visual question answering is a vision-and-language multimodal task, that aims at predicting answers given samples from the question and image modalities. Most recent methods focus on learning a good joint embedding space of images and questions, either by improving the interaction between these two modalities, or by making it a more discriminant space. However, how informative this joint space is, has not been well explored. In this paper, we propose a novel regularization for VQA models, Constrained Optimization using Barlow's theory (COB), that improves the information content of the joint space by minimizing the redundancy. It reduces the correlation between the learned feature components and thereby disentangles semantic concepts. Our model also aligns the joint space with the answer embedding space, where we consider the answer and image+question as two different `views' of what in essence is the same semantic information. We propose a constrained optimization policy to balance the categorical and redundancy minimization forces. When built on the state-of-the-art GGE model, the resulting model improves VQA accuracy by 1.4% and 4% on the VQA-CP v2 and VQA v2 datasets respectively. The model also exhibits better interpretability.