UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models

TL;DR

UniE2F introduces a unified diffusion-based framework that reconstructs high-fidelity video frames from sparse event camera data, extending to interpolation and prediction, and significantly outperforms prior methods.

Contribution

The paper presents a novel unified diffusion model that reconstructs, interpolates, and predicts video frames from event data, leveraging a pre-trained video diffusion prior and event-based residual guidance.

Findings

Outperforms previous methods quantitatively and qualitatively

Effective zero-shot frame interpolation and prediction

High-fidelity video reconstruction from sparse event data

Abstract

Event cameras excel at high-speed, low-power, and high-dynamic-range scene perception. However, as they fundamentally record only relative intensity changes rather than absolute intensity, the resulting data streams suffer from a significant loss of spatial information and static texture details. In this paper, we address this limitation by leveraging the generative prior of a pre-trained video diffusion model to reconstruct high-fidelity video frames from sparse event data. Specifically, we first establish a baseline model by directly applying event data as a condition to synthesize videos. Then, based on the physical correlation between the event stream and video frames, we further introduce the event-based inter-frame residual guidance to enhance the accuracy of video frame reconstruction. Furthermore, we extend our method to video frame interpolation and prediction in a zero-shot manner by modulating the reverse diffusion sampling process, thereby creating a unified event-to-frame reconstruction framework. Experimental results on real-world and synthetic datasets demonstrate that our method significantly outperforms previous approaches both quantitatively and qualitatively. We also refer the reviewers to the video demo contained in the supplementary material for video results. The code will be publicly available at https://github.com/CS-GangXu/UniE2F.