A Prototype Unit for Image De-raining using Time-Lapse Data

TL;DR

This paper introduces the Rain Streak Prototype Unit (RsPU), a memory-efficient module that encodes rain streak features from time-lapse data to improve single-image de-raining performance.

Contribution

The novel RsPU encodes rain streak features as real-time prototypes, reducing memory use and integrating with encoder-decoder networks for effective de-raining.

Findings

Achieves competitive results on multiple benchmarks

Reduces memory consumption compared to existing methods

Demonstrates effectiveness through ablation studies

Abstract

We address the challenge of single-image de-raining, a task that involves recovering rain-free background information from a single rain image. While recent advancements have utilized real-world time-lapse data for training, enabling the estimation of consistent backgrounds and realistic rain streaks, these methods often suffer from computational and memory consumption, limiting their applicability in real-world scenarios. In this paper, we introduce a novel solution: the Rain Streak Prototype Unit (RsPU). The RsPU efficiently encodes rain streak-relevant features as real-time prototypes derived from time-lapse data, eliminating the need for excessive memory resources. Our de-raining network combines encoder-decoder networks with the RsPU, allowing us to learn and encapsulate diverse rain streak-relevant features as concise prototypes, employing an attention-based approach. To ensure the effectiveness of our approach, we propose a feature prototype loss encompassing cohesion and divergence components. This loss function captures both the compactness and diversity aspects of the prototypical rain streak features within the RsPU. Our method evaluates various de-raining benchmarks, accompanied by comprehensive ablation studies. We show that it can achieve competitive results in various rain images compared to state-of-the-art methods.