NEWSKVQA: Knowledge-Aware News Video Question Answering

TL;DR

This paper introduces NEWSKVQA, a new dataset and approach for knowledge-based question answering in news videos, enabling better understanding and retrieval of news content through multimodal reasoning.

Contribution

The paper presents a large-scale news video dataset with questions and a novel multimodal reasoning method for knowledge-aware video question answering.

Findings

The dataset contains 12K videos and 1M questions.

The proposed method achieves strong baseline performance.

The dataset is publicly available for research use.

Abstract

Answering questions in the context of videos can be helpful in video indexing, video retrieval systems, video summarization, learning management systems and surveillance video analysis. Although there exists a large body of work on visual question answering, work on video question answering (1) is limited to domains like movies, TV shows, gameplay, or human activity, and (2) is mostly based on common sense reasoning. In this paper, we explore a new frontier in video question answering: answering knowledge-based questions in the context of news videos. To this end, we curate a new dataset of 12K news videos spanning across 156 hours with 1M multiple-choice question-answer pairs covering 8263 unique entities. We make the dataset publicly available. Using this dataset, we propose a novel approach, NEWSKVQA (Knowledge-Aware News Video Question Answering) which performs multi-modal inferencing over textual multiple-choice questions, videos, their transcripts and knowledge base, and presents a strong baseline.