Self-supervised pre-training and contrastive representation learning for multiple-choice video QA

TL;DR

This paper introduces a novel self-supervised pre-training and contrastive learning framework for multiple-choice video QA, improving model performance by leveraging broader contextual understanding and focused attention mechanisms.

Contribution

The paper proposes a new training scheme combining self-supervised pre-training and supervised contrastive learning for video QA, enhancing understanding without additional annotations.

Findings

Achieved state-of-the-art results on TVQA, TVQA+, and DramaQA datasets.

Demonstrated effectiveness of contrastive learning with masking noise.

Validated the benefit of locally aligned attention for relevant frame focus.

Abstract

Video Question Answering (Video QA) requires fine-grained understanding of both video and language modalities to answer the given questions. In this paper, we propose novel training schemes for multiple-choice video question answering with a self-supervised pre-training stage and a supervised contrastive learning in the main stage as an auxiliary learning. In the self-supervised pre-training stage, we transform the original problem format of predicting the correct answer into the one that predicts the relevant question to provide a model with broader contextual inputs without any further dataset or annotation. For contrastive learning in the main stage, we add a masking noise to the input corresponding to the ground-truth answer, and consider the original input of the ground-truth answer as a positive sample, while treating the rest as negative samples. By mapping the positive sample closer to the masked input, we show that the model performance is improved. We further employ locally aligned attention to focus more effectively on the video frames that are particularly relevant to the given corresponding subtitle sentences. We evaluate our proposed model on highly competitive benchmark datasets related to multiple-choice video QA: TVQA, TVQA+, and DramaQA. Experimental results show that our model achieves state-of-the-art performance on all datasets. We also validate our approaches through further analyses.