A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot
Aanisha Bhattacharya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn, Shah, Changyou Chen

TL;DR
This paper introduces a zero-shot video understanding approach by verbalizing videos into natural language descriptions, enabling better performance on multiple tasks without requiring annotated datasets.
Contribution
It proposes a novel method of verbalizing videos to leverage large language models for zero-shot understanding, outperforming supervised baselines across various tasks.
Findings
Zero-shot method outperforms supervised baselines
Verbalization enables effective multi-task video understanding
Public dataset released for persuasion strategy identification
Abstract
Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities. They incorporate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey meaning. There is a dearth of large annotated training datasets in the multimedia domain hindering the development of supervised learning models with satisfactory performance for real-world applications. On the other hand, the rise of large language models (LLMs) has witnessed remarkable zero-shot performance in various natural language processing (NLP) tasks, such as emotion classification, question-answering, and topic classification. To leverage such advanced techniques to bridge this performance gap in multimedia understanding, we propose verbalizing long videos to generate their descriptions in natural language,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Misinformation and Its Impacts
