Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras

TL;DR

This paper introduces Talk2Event, a large-scale benchmark and a novel grounding framework for understanding dynamic scenes from event camera data, bridging perception and language in real-world environments.

Contribution

It presents the first large-scale dataset with language annotations for event-based perception and proposes EventRefer, a multi-attribute grounding model that fuses diverse cues for improved scene understanding.

Findings

EventRefer outperforms state-of-the-art baselines across multiple settings.

The dataset enables research in language-driven perception from event data.

The approach effectively integrates appearance, status, and relational cues.

Abstract

Event cameras offer microsecond-level latency and robustness to motion blur, making them ideal for understanding dynamic environments. Yet, connecting these asynchronous streams to human language remains an open challenge. We introduce Talk2Event, the first large-scale benchmark for language-driven object grounding in event-based perception. Built from real-world driving data, we provide over 30,000 validated referring expressions, each enriched with four grounding attributes -- appearance, status, relation to viewer, and relation to other objects -- bridging spatial, temporal, and relational reasoning. To fully exploit these cues, we propose EventRefer, an attribute-aware grounding framework that dynamically fuses multi-attribute representations through a Mixture of Event-Attribute Experts (MoEE). Our method adapts to different modalities and scene dynamics, achieving consistent gains over state-of-the-art baselines in event-only, frame-only, and event-frame fusion settings. We hope our dataset and approach will establish a foundation for advancing multimodal, temporally-aware, and language-driven perception in real-world robotics and autonomy.