IE2Video: Adapting Pretrained Diffusion Models for Event-Based Video Reconstruction

TL;DR

This paper introduces IE2Video, a hybrid event-based video reconstruction method that combines sparse RGB keyframes with event streams, utilizing diffusion models to produce high-quality videos with lower power consumption.

Contribution

It proposes a novel task of reconstructing RGB videos from a single frame and event data, and explores two architectural strategies including a diffusion-based approach that outperforms autoregressive models.

Findings

Diffusion-based approach achieves 33% better perceptual quality than autoregressive baseline.

Method demonstrates robust cross-dataset generalization across multiple event camera datasets.

Reconstruction maintains high quality over varying sequence lengths and unseen configurations.

Abstract

Continuous video monitoring in surveillance, robotics, and wearable systems faces a fundamental power constraint: conventional RGB cameras consume substantial energy through fixed-rate capture. Event cameras offer sparse, motion-driven sensing with low power consumption, but produce asynchronous event streams rather than RGB video. We propose a hybrid capture paradigm that records sparse RGB keyframes alongside continuous event streams, then reconstructs full RGB video offline -- reducing capture power consumption while maintaining standard video output for downstream applications. We introduce the Image and Event to Video (IE2Video) task: reconstructing RGB video sequences from a single initial frame and subsequent event camera data. We investigate two architectural strategies: adapting an autoregressive model (HyperE2VID) for RGB generation, and injecting event representations into a pretrained text-to-video diffusion model (LTX) via learned encoders and low-rank adaptation. Our experiments demonstrate that the diffusion-based approach achieves 33\% better perceptual quality than the autoregressive baseline (0.283 vs 0.422 LPIPS). We validate our approach across three event camera datasets (BS-ERGB, HS-ERGB far/close) at varying sequence lengths (32-128 frames), demonstrating robust cross-dataset generalization with strong performance on unseen capture configurations.