Logically Consistent Loss for Visual Question Answering

TL;DR

This paper introduces a logic-based loss function and data organization techniques to improve the logical consistency and performance of neural network models in visual question answering tasks.

Contribution

It proposes a model-agnostic logically consistent loss and data organization methods that enhance answer consistency and accuracy in VQA systems.

Findings

Increased answer consistency in VQA models.

Improved performance with the proposed loss and data organization.

Applicable to various QA models beyond MAC-net.

Abstract

Given an image, a back-ground knowledge, and a set of questions about an object, human learners answer the questions very consistently regardless of question forms and semantic tasks. The current advancement in neural-network based Visual Question Answering (VQA), despite their impressive performance, cannot ensure such consistency due to identically distribution (i.i.d.) assumption. We propose a new model-agnostic logic constraint to tackle this issue by formulating a logically consistent loss in the multi-task learning framework as well as a data organisation called family-batch and hybrid-batch. To demonstrate usefulness of this proposal, we train and evaluate MAC-net based VQA machines with and without the proposed logically consistent loss and the proposed data organization. The experiments confirm that the proposed loss formulae and introduction of hybrid-batch leads to more consistency as well as better performance. Though the proposed approach is tested with MAC-net, it can be utilised in any other QA methods whenever the logical consistency between answers exist.