Grounding Partially-Defined Events in Multimodal Data

TL;DR

This paper introduces a new multimodal framework and benchmark for extracting and understanding partially-defined events from unstructured video data using large language models, highlighting challenges and potential in event-centric video-language systems.

Contribution

It presents a novel multimodal formulation for partially-defined event extraction, a new benchmark dataset MultiVENT-G, and evaluates LLM-driven approaches for complex event understanding.

Findings

LLMs show promise in multimodal event analysis

Event understanding remains a challenging task

Benchmark MultiVENT-G provides a new resource for research

Abstract

How are we able to learn about complex current events just from short snippets of video? While natural language enables straightforward ways to represent under-specified, partially observable events, visual data does not facilitate analogous methods and, consequently, introduces unique challenges in event understanding. With the growing prevalence of vision-capable AI agents, these systems must be able to model events from collections of unstructured video data. To tackle robust event modeling in multimodal settings, we introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a corresponding benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. We propose a collection of LLM-driven approaches to the task of multimodal event analysis, and evaluate them on MultiVENT-G. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.