Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

TL;DR

Event-MLLM enhances multimodal large language models to perform vision-language reasoning under extreme illumination by fusing event streams with RGB inputs and using adaptive illumination-aware components.

Contribution

The paper introduces a novel event-enhanced multimodal model with adaptive fusion and illumination correction, along with a new multi-illumination event-instruction dataset.

Findings

Event-MLLM outperforms existing models in extreme lighting conditions.

The model achieves state-of-the-art results in robust perception and reasoning.

A new dataset with 2,241 samples across 17 brightness levels supports evaluation.

Abstract

Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.