TL;DR
This paper introduces a scalable method for video question answering by automatically generating large datasets from narrated videos using cross-modal supervision and a question generation transformer, enabling zero-shot learning and outperforming existing methods.
Contribution
The authors propose a novel automatic dataset creation process for VideoQA using question generation from narrations, and introduce a new zero-shot VideoQA task with improved performance.
Findings
Generated the 69M question-answer triplet dataset from narrated videos.
Achieved state-of-the-art results on multiple VideoQA benchmarks.
Demonstrated strong zero-shot performance, especially on rare answers.
Abstract
Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and show excellent…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
