Are we asking the right questions in MovieQA?

TL;DR

This paper investigates biases in the MovieQA dataset and demonstrates that a simple question-answering model, leveraging proper word embeddings, can answer many questions without visual or narrative context, raising questions about dataset validity.

Contribution

The paper reveals significant biases in MovieQA and shows that a straightforward model with appropriate embeddings can outperform complex models by exploiting these biases.

Findings

Approximately 50% of questions can be answered using only questions and answers.

Using proper word embeddings is crucial for model performance.

Simple models can outperform complex ones by exploiting dataset biases.

Abstract

Joint vision and language tasks like visual question answering are fascinating because they explore high-level understanding, but at the same time, can be more prone to language biases. In this paper, we explore the biases in the MovieQA dataset and propose a strikingly simple model which can exploit them. We find that using the right word embedding is of utmost importance. By using an appropriately trained word embedding, about half the Question-Answers (QAs) can be answered by looking at the questions and answers alone, completely ignoring narrative context from video clips, subtitles, and movie scripts. Compared to the best published papers on the leaderboard, our simple question + answer only model improves accuracy by 5% for video + subtitle category, 5% for subtitle, 15% for DVS and 6% higher for scripts.