EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs

TL;DR

EventBench is a comprehensive benchmark for evaluating event-based multimodal large language models, featuring diverse tasks, large-scale datasets, and evaluation of both open and closed-source models to identify current strengths and weaknesses.

Contribution

The paper introduces EventBench, a novel, open, and large-scale benchmark with diverse tasks and datasets for comprehensive assessment of event-based MLLMs.

Findings

Current models excel in event stream understanding.

Models still struggle with fine-grained recognition.

Spatial reasoning remains challenging for existing models.

Abstract

Multimodal large language models (MLLMs) have made significant advancements in event-based vision, yet the comprehensive evaluation of their capabilities within a unified benchmark remains largely unexplored. In this work, we introduce EventBench, a benchmark that offers eight diverse task metrics together with a large-scale event stream dataset. EventBench differs from existing event-based benchmarks in four key aspects: (1) openness in accessibility, releasing all raw event streams and task instructions across eight evaluation metrics; (2) diversity in task coverage, spanning understanding, recognition, and spatial reasoning tasks for comprehensive capability assessment; (3) integration in spatial dimensions, pioneering the design of 3D spatial reasoning tasks for event-based MLLMs; and (4) scale in data volume, with an accompanying training set of over one million event-text pairs supporting large-scale training and evaluation. Using EventBench, we evaluate state-of-the-art closed-source models such as GPT-5 and Gemini-2.5 Pro, leading open-source models including Qwen2.5-VL and InternVL3, and event-based MLLMs such as EventGPT that directly process raw event streams. Extensive evaluation reveals that while current event-based MLLMs demonstrate strong performance in event stream understanding, they continue to struggle with fine-grained recognition and spatial reasoning.