TPSeNCE: Towards Artifact-Free Realistic Rain Generation for Deraining and Object Detection in Rain

TL;DR

This paper introduces TPSeNCE, a novel unpaired image translation framework that generates realistic rainy images with minimal artifacts, improving deraining and object detection, and extends to snowy and night images.

Contribution

The paper proposes a new TPS constraint and SeNCE strategy for artifact-free rain generation, enhancing scene understanding in adverse weather conditions.

Findings

Realistic rain generation with minimal artifacts demonstrated.

Improved deraining and object detection performance.

Framework applicable to snowy and night images.

Abstract

Rain generation algorithms have the potential to improve the generalization of deraining methods and scene understanding in rainy conditions. However, in practice, they produce artifacts and distortions and struggle to control the amount of rain generated due to a lack of proper constraints. In this paper, we propose an unpaired image-to-image translation framework for generating realistic rainy images. We first introduce a Triangular Probability Similarity (TPS) constraint to guide the generated images toward clear and rainy images in the discriminator manifold, thereby minimizing artifacts and distortions during rain generation. Unlike conventional contrastive learning approaches, which indiscriminately push negative samples away from the anchors, we propose a Semantic Noise Contrastive Estimation (SeNCE) strategy and reassess the pushing force of negative samples based on the semantic similarity between the clear and the rainy images and the feature similarity between the anchor and the negative samples. Experiments demonstrate realistic rain generation with minimal artifacts and distortions, which benefits image deraining and object detection in rain. Furthermore, the method can be used to generate realistic snowy and night images, underscoring its potential for broader applicability. Code is available at https://github.com/ShenZheng2000/TPSeNCE.