MultiVENT: Multilingual Videos of Events with Aligned Natural Text

TL;DR

MultiVENT is a new multilingual, event-centric video dataset that includes both professional and non-professional footage, designed to enhance models' ability to understand and retrieve diverse online news videos across multiple languages.

Contribution

This paper introduces MultiVENT, the first multilingual, event-focused video dataset with aligned text, covering diverse news formats and languages, to improve multimodal and multilingual news understanding.

Findings

MultiVENT covers five languages and includes both broadcast and non-professional videos.

A baseline multilingual video retrieval model demonstrates the dataset's utility.

Analysis shows the potential for improved factually accurate, multilingual news models.

Abstract

Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage. Datasets that reflect the diverse array of multimodal, multilingual news sources available online could be used to teach models to benefit from this shift, but existing news video datasets focus on traditional news broadcasts produced for English-speaking audiences. We address this limitation by constructing MultiVENT, a dataset of multilingual, event-centric videos grounded in text documents across five target languages. MultiVENT includes both news broadcast videos and non-professional event footage, which we use to analyze the state of online news videos and how they can be leveraged to build robust, factually accurate models. Finally, we provide a model for complex, multilingual video retrieval to serve as a baseline for information retrieval using MultiVENT.