Photorealistic Image Reconstruction from Hybrid Intensity and Event based Sensor

TL;DR

This paper introduces a method to generate photorealistic images by combining low frame rate camera images with high-temporal-rate event sensor data, overcoming previous limitations of non-photorealistic reconstructions.

Contribution

The authors propose a novel hybrid approach that uses depth and ego-motion estimation to warp low frame rate images into dense event data, producing more realistic images than prior methods.

Findings

Produces more photorealistic images than previous algorithms

Robust to abrupt camera movements

Effective even with noisy event data

Abstract

Event sensors output a stream of asynchronous brightness changes (called ``events'') at a very high temporal rate. Previous works on recovering the lost intensity information from the event sensor data have heavily relied on the event stream, which makes the reconstructed images non-photorealistic and also susceptible to noise in the event stream. We propose to reconstruct photorealistic intensity images from a hybrid sensor consisting of a low frame rate conventional camera, which has the scene texture information, along with the event sensor. To accomplish our task, we warp the low frame rate intensity images to temporally dense locations of the event data by estimating a spatially dense scene depth and temporally dense sensor ego-motion. The results obtained from our algorithm are more photorealistic compared to any of the previous state-of-the-art algorithms. We also demonstrate our algorithm's robustness to abrupt camera motion and noise in the event sensor data.