Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation

TL;DR

This paper proposes a knowledge distillation approach to handle missing answer modalities in visual question answering difference prediction, improving performance on VizWiz and VQA-V2 datasets.

Contribution

It introduces a novel privileged knowledge distillation scheme and a 'Big' Teacher model to effectively address missing answer modalities in VQA difference prediction.

Findings

The method outperforms baseline models on VizWiz and VQA-V2 datasets.

Knowledge distillation improves prediction accuracy with missing answer modalities.

Extensive experiments validate the effectiveness of the proposed approach.

Abstract

In this work, we address the issues of missing modalities that have arisen from the Visual Question Answer-Difference prediction task and find a novel method to solve the task at hand. We address the missing modality-the ground truth answers-that are not present at test time and use a privileged knowledge distillation scheme to deal with the issue of the missing modality. In order to efficiently do so, we first introduce a model, the "Big" Teacher, that takes the image/question/answer triplet as its input and outperforms the baseline, then use a combination of models to distill knowledge to a target network (student) that only takes the image/question pair as its inputs. We experiment our models on the VizWiz and VQA-V2 Answer Difference datasets and show through extensive experimentation and ablation the performances of our method and a diverse possibility for future research.