E2VIDiff: Perceptual Events-to-Video Reconstruction using Diffusion Priors

TL;DR

This paper introduces a diffusion-based approach for converting event camera data into realistic, colorful videos, significantly improving perceptual quality over traditional methods by leveraging pretrained diffusion models.

Contribution

The paper presents a novel diffusion model framework for events-to-video reconstruction, enhancing perceptual realism and diversity compared to prior regression-based methods.

Findings

Produces diverse, realistic frames faithful to event data

Achieves better perception-distortion trade-off than previous methods

Demonstrates superior results on benchmark datasets

Abstract

Event cameras, mimicking the human retina, capture brightness changes with unparalleled temporal resolution and dynamic range. Integrating events into intensities poses a highly ill-posed challenge, marred by initial condition ambiguities. Traditional regression-based deep learning methods fall short in perceptual quality, offering deterministic and often unrealistic reconstructions. In this paper, we introduce diffusion models to events-to-video reconstruction, achieving colorful, realistic, and perceptually superior video generation from achromatic events. Powered by the image generation ability and knowledge of pretrained diffusion models, the proposed method can achieve a better trade-off between the perception and distortion of the reconstructed frame compared to previous solutions. Extensive experiments on benchmark datasets demonstrate that our approach can produce diverse, realistic frames with faithfulness to the given events.