High Frame Rate Video Reconstruction based on an Event Camera

TL;DR

This paper introduces a novel event-based model for reconstructing high frame rate, sharp videos from blurred images and event data captured by event cameras, significantly improving video clarity and temporal resolution.

Contribution

The authors propose the Event-based Double Integral (EDI) and multi-EDI models, along with an efficient solver, to effectively deblur and enhance video frame rate using event camera data.

Findings

Superior deblurring performance on synthetic and real datasets.

Enhanced temporal resolution in reconstructed videos.

Outperforms state-of-the-art methods in quality and efficiency.

Abstract

Event-based cameras measure intensity changes (called `events') with microsecond accuracy under high-speed motion and challenging lighting conditions. With the `active pixel sensor' (APS), the `Dynamic and Active-pixel Vision Sensor' (DAVIS) allows the simultaneous output of intensity frames and events. However, the output images are captured at a relatively low frame rate and often suffer from motion blur. A blurred image can be regarded as the integral of a sequence of latent images, while events indicate changes between the latent images. Thus, we are able to model the blur-generation process by associating event data to a latent sharp image. Based on the abundant event data alongside a low frame rate, easily blurred images, we propose a simple yet effective approach to reconstruct high-quality and high frame rate sharp videos. Starting with a single blurred frame and its event data from DAVIS, we propose the Event-based Double Integral (EDI) model and solve it by adding regularization terms. Then, we extend it to multiple Event-based Double Integral (mEDI) model to get more smooth results based on multiple images and their events. Furthermore, we provide a new and more efficient solver to minimize the proposed energy model. By optimizing the energy function, we achieve significant improvements in removing blur and the reconstruction of a high temporal resolution video. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Experimental results on both synthetic and real datasets demonstrate the superiority of our mEDI model and optimization method compared to the state-of-the-art.