Uncovering Temporal Context for Video Question and Answering

TL;DR

This paper proposes a novel encoder-decoder neural network approach for video question answering that captures temporal context to understand past, present, and future events, significantly improving performance on large-scale datasets.

Contribution

It introduces a dual-channel ranking loss and a new dataset for temporal video question answering, advancing the understanding of video content over time.

Findings

Outperforms baseline methods on large-scale datasets

Effectively models temporal structures in videos

Achieves significant accuracy improvements

Abstract

In this work, we introduce Video Question Answering in temporal domain to infer the past, describe the present and predict the future. We present an encoder-decoder approach using Recurrent Neural Networks to learn temporal structures of videos and introduce a dual-channel ranking loss to answer multiple-choice questions. We explore approaches for finer understanding of video content using question form of "fill-in-the-blank", and managed to collect 109,895 video clips with duration over 1,000 hours from TACoS, MPII-MD, MEDTest 14 datasets, while the corresponding 390,744 questions are generated from annotations. Extensive experiments demonstrate that our approach significantly outperforms the compared baselines.