Generating Video Descriptions with Topic Guidance
Shizhe Chen, Jia Chen, Qin Jin

TL;DR
This paper introduces a novel topic-guided model for video captioning that leverages both predefined and data-driven topics, significantly improving description quality on large datasets by incorporating topic information into the caption generation process.
Contribution
The paper proposes a new topic-guided video captioning model that uses unsupervised topic mining and multi-modal video features to enhance caption relevance and accuracy.
Findings
Outperforms previous models on MSR-VTT dataset
Effectively incorporates topic information into caption generation
Achieves state-of-the-art results in video captioning challenge
Abstract
Generating video descriptions in natural language (a.k.a. video captioning) is a more challenging task than image captioning as the videos are intrinsically more complicated than images in two aspects. First, videos cover a broader range of topics, such as news, music, sports and so on. Second, multiple topics could coexist in the same video. In this paper, we propose a novel caption model, topic-guided model (TGM), to generate topic-oriented descriptions for videos in the wild via exploiting topic information. In addition to predefined topics, i.e., category tags crawled from the web, we also mine topics in a data-driven way based on training captions by an unsupervised topic mining model. We show that data-driven topics reflect a better topic schema than the predefined topics. As for testing video topic prediction, we treat the topic mining model as teacher to train the student, the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Video Analysis and Summarization · Natural Language Processing Techniques
