EventGPT: Event Stream Understanding with Multimodal Large Language Models

TL;DR

EventGPT is a pioneering multimodal large language model designed specifically for understanding event streams from event cameras, addressing domain gaps through a three-stage training process and achieving superior performance in scene understanding tasks.

Contribution

This work introduces the first MLLM for event stream understanding, developing a novel three-stage optimization paradigm to bridge domain gaps and enhance event-based scene comprehension.

Findings

EventGPT outperforms previous models in generation quality.

It achieves higher descriptive accuracy in event scene understanding.

The model demonstrates improved reasoning capabilities on benchmarks.

Abstract

Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.