TVQA: Localized, Compositional Video Question Answering

TL;DR

TVQA introduces a large-scale, compositional video question-answering dataset based on TV show clips, challenging models to localize moments, understand dialogue, and recognize visual concepts.

Contribution

The paper presents TVQA, a new dataset for video QA with complex questions requiring localization, dialogue comprehension, and visual recognition, along with baseline models.

Findings

Baseline models achieve moderate performance on TVQA.

Analysis reveals challenges in localization and dialogue understanding.

The dataset enables research on multi-modal, compositional video reasoning.

Abstract

Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.