HDR Reconstruction from Bracketed Exposures and Events

TL;DR

This paper introduces a novel multi-modal HDR reconstruction system that combines bracketed LDR images with event-based camera data, significantly improving HDR quality especially in dynamic scenes.

Contribution

It presents an end-to-end learning framework that fuses image and event data using attention and alignment modules, including a new event-to-image feature distillation technique.

Findings

Outperforms state-of-the-art multi-frame HDR methods on synthetic and real data.

Achieves 2dB and 1dB higher PSNR-L and PSNR-mu scores on HdM HDR dataset.

Effectively leverages high temporal resolution of events for dynamic scene HDR reconstruction.

Abstract

Reconstruction of high-quality HDR images is at the core of modern computational photography. Significant progress has been made with multi-frame HDR reconstruction methods, producing high-resolution, rich and accurate color reconstructions with high-frequency details. However, they are still prone to fail in dynamic or largely over-exposed scenes, where frame misalignment often results in visible ghosting artifacts. Recent approaches attempt to alleviate this by utilizing an event-based camera (EBC), which measures only binary changes of illuminations. Despite their desirable high temporal resolution and dynamic range characteristics, such approaches have not outperformed traditional multi-frame reconstruction methods, mainly due to the lack of color information and low-resolution sensors. In this paper, we propose to leverage both bracketed LDR images and simultaneously captured events to obtain the best of both worlds: high-quality RGB information from bracketed LDRs and complementary high frequency and dynamic range information from events. We present a multi-modal end-to-end learning-based HDR imaging system that fuses bracketed images and event modalities in the feature domain using attention and multi-scale spatial alignment modules. We propose a novel event-to-image feature distillation module that learns to translate event features into the image-feature space with self-supervision. Our framework exploits the higher temporal resolution of events by sub-sampling the input event streams using a sliding window, enriching our combined feature representation. Our proposed approach surpasses SoTA multi-frame HDR reconstruction methods using synthetic and real events, with a 2dB and 1dB improvement in PSNR-L and PSNR-mu on the HdM HDR dataset, respectively.