EGVD: Event-Guided Video Deraining

TL;DR

This paper introduces EGVD, a novel event-guided neural network that leverages neuromorphic event camera data to improve video deraining performance, especially in complex scenes with dynamic motion and lighting.

Contribution

The paper presents an end-to-end learning framework utilizing event camera data, including an event-aware motion detection module and a pyramidal adaptive selection module, for enhanced video deraining.

Findings

Outperforms state-of-the-art methods on synthetic datasets

Demonstrates effectiveness on real-world rainy videos with event data

Provides a new dataset with synchronized rainy videos and event streams

Abstract

With the rapid development of deep learning, video deraining has experienced significant progress. However, existing video deraining pipelines cannot achieve satisfying performance for scenes with rain layers of complex spatio-temporal distribution. In this paper, we approach video deraining by employing an event camera. As a neuromorphic sensor, the event camera suits scenes of non-uniform motion and dynamic light conditions. We propose an end-to-end learning-based network to unlock the potential of the event camera for video deraining. First, we devise an event-aware motion detection module to adaptively aggregate multi-frame motion contexts using event-aware masks. Second, we design a pyramidal adaptive selection module for reliably separating the background and rain layers by incorporating multi-modal contextualized priors. In addition, we build a real-world dataset consisting of rainy videos and temporally synchronized event streams. We compare our method with extensive state-of-the-art methods on synthetic and self-collected real-world datasets, demonstrating the clear superiority of our method. The code and dataset are available at \url{https://github.com/booker-max/EGVD}.