Towards Better De-raining Generalization via Rainy Characteristics Memorization and Replay

TL;DR

This paper proposes a continual learning framework for image de-raining that mimics human memory mechanisms, enabling networks to progressively learn from multiple datasets and improve generalization to diverse real-world rainy conditions.

Contribution

It introduces a novel framework combining GANs, replay, and knowledge distillation to enhance de-raining models' ability to generalize across varied datasets.

Findings

Outperforms state-of-the-art methods in generalization.

Enables continuous knowledge accumulation across multiple datasets.

Improves performance on unseen rainy scenes.

Abstract

Current image de-raining methods primarily learn from a limited dataset, leading to inadequate performance in varied real-world rainy conditions. To tackle this, we introduce a new framework that enables networks to progressively expand their de-raining knowledge base by tapping into a growing pool of datasets, significantly boosting their adaptability. Drawing inspiration from the human brain's ability to continuously absorb and generalize from ongoing experiences, our approach borrow the mechanism of the complementary learning system. Specifically, we first deploy Generative Adversarial Networks (GANs) to capture and retain the unique features of new data, mirroring the hippocampus's role in learning and memory. Then, the de-raining network is trained with both existing and GAN-synthesized data, mimicking the process of hippocampal replay and interleaved learning. Furthermore, we employ knowledge distillation with the replayed data to replicate the synergy between the neocortex's activity patterns triggered by hippocampal replays and the pre-existing neocortical knowledge. This comprehensive framework empowers the de-raining network to amass knowledge from various datasets, continually enhancing its performance on previously unseen rainy scenes. Our testing on three benchmark de-raining networks confirms the framework's effectiveness. It not only facilitates continuous knowledge accumulation across six datasets but also surpasses state-of-the-art methods in generalizing to new real-world scenarios.