Learning from Rendering: Realistic and Controllable Extreme Rainy Image Synthesis for Autonomous Driving Simulation

TL;DR

This paper introduces CARLARain, a novel extreme rainy image synthesizer that combines rendering realism with learning-based controllability, significantly improving semantic segmentation accuracy in rainy driving scenes.

Contribution

We propose a learning-from-rendering rainy image synthesizer integrated with CARLA, enabling realistic and controllable extreme rain simulation for autonomous driving evaluation.

Findings

Improved semantic segmentation accuracy by 5-8% mIoU on synthetic data.

Effective enhancement of model performance in real extreme rainy scenarios.

CARLARain provides high-quality paired rainy and clean images under complex illumination.

Abstract

Autonomous driving simulators provide an effective and low-cost alternative for evaluating or enhancing visual perception models. However, the reliability of evaluation depends on the diversity and realism of the generated scenes. Extreme weather conditions, particularly extreme rainfalls, are rare and costly to capture in real-world settings. While simulated environments can help address this limitation, existing rainy image synthesizers often suffer from poor controllability over illumination and limited realism, which significantly undermines the effectiveness of the model evaluation. To that end, we propose a learning-from-rendering rainy image synthesizer, which combines the benefits of the realism of rendering-based methods and the controllability of learning-based methods. To validate the effectiveness of our extreme rainy image synthesizer on semantic segmentation task, we require a continuous set of well-labeled extreme rainy images. By integrating the proposed synthesizer with the CARLA driving simulator, we develop CARLARain an extreme rainy street scene simulator which can obtain paired rainy-clean images and labels under complex illumination conditions. Qualitative and quantitative experiments validate that CARLARain can effectively improve the accuracy of semantic segmentation models in extreme rainy scenes, with the models' accuracy (mIoU) improved by 5% - 8% on the synthetic dataset and significantly enhanced in real extreme rainy scenarios under complex illuminations. Our source code and datasets are available at https://github.com/kb824999404/CARLARain/.