Event-based Image Deblurring with Dynamic Motion Awareness

TL;DR

This paper introduces a novel event-based image deblurring method that leverages dynamic motion awareness through modulated deformable convolutions and a multi-scale approach, demonstrating improved robustness and performance on synthetic and real data.

Contribution

It proposes a divide-and-conquer approach using dynamic modulated deformable convolutions for motion encoding and introduces the first dataset with real RGB blur images and events.

Findings

Improved PSNR by up to 1.57dB on synthetic data

Enhanced robustness with event integration

First dataset with real RGB blur and event pairs

Abstract

Non-uniform image deblurring is a challenging task due to the lack of temporal and textural information in the blurry image itself. Complementary information from auxiliary sensors such event sensors are being explored to address these limitations. The latter can record changes in a logarithmic intensity asynchronously, called events, with high temporal resolution and high dynamic range. Current event-based deblurring methods combine the blurry image with events to jointly estimate per-pixel motion and the deblur operator. In this paper, we argue that a divide-and-conquer approach is more suitable for this task. To this end, we propose to use modulated deformable convolutions, whose kernel offsets and modulation masks are dynamically estimated from events to encode the motion in the scene, while the deblur operator is learned from the combination of blurry image and corresponding events. Furthermore, we employ a coarse-to-fine multi-scale reconstruction approach to cope with the inherent sparsity of events in low contrast regions. Importantly, we introduce the first dataset containing pairs of real RGB blur images and related events during the exposure time. Our results show better overall robustness when using events, with improvements in PSNR by up to 1.57dB on synthetic data and 1.08 dB on real event data.