EventHDR: from Event to High-Speed HDR Videos and Beyond

TL;DR

This paper introduces EventHDR, a neural network-based method for reconstructing high-speed HDR videos from event camera data, supported by a new real-world dataset, improving quality and application scope in event-based vision tasks.

Contribution

The paper presents a recurrent CNN with key frame guidance for event-to-HDR video reconstruction and introduces the first real paired dataset for this task.

Findings

High-quality, high-speed HDR videos are reconstructed successfully.

The method outperforms previous approaches with fewer artifacts.

The dataset enables more accurate and realistic event-to-HDR research.

Abstract

Event cameras are innovative neuromorphic sensors that asynchronously capture the scene dynamics. Due to the event-triggering mechanism, such cameras record event streams with much shorter response latency and higher intensity sensitivity compared to conventional cameras. On the basis of these features, previous works have attempted to reconstruct high dynamic range (HDR) videos from events, but have either suffered from unrealistic artifacts or failed to provide sufficiently high frame rates. In this paper, we present a recurrent convolutional neural network that reconstruct high-speed HDR videos from event sequences, with a key frame guidance to prevent potential error accumulation caused by the sparse event data. Additionally, to address the problem of severely limited real dataset, we develop a new optical system to collect a real-world dataset with paired high-speed HDR videos and event streams, facilitating future research in this field. Our dataset provides the first real paired dataset for event-to-HDR reconstruction, avoiding potential inaccuracies from simulation strategies. Experimental results demonstrate that our method can generate high-quality, high-speed HDR videos. We further explore the potential of our work in cross-camera reconstruction and downstream computer vision tasks, including object detection, panoramic segmentation, optical flow estimation, and monocular depth estimation under HDR scenarios.