IntrinsicWeather: Controllable Weather Editing in Intrinsic Space

TL;DR

IntrinsicWeather is a diffusion-based framework enabling controllable weather editing in images through intrinsic maps and text prompts, improving spatial accuracy and robustness in outdoor scene editing.

Contribution

It introduces a novel intrinsic map-aware diffusion framework with a new attention mechanism and a large annotated dataset for weather editing in outdoor scenes.

Findings

Outperforms state-of-the-art pixel-space editing methods.

Enables fine-grained weather control via CLIP-space interpolation.

Enhances robustness of detection and segmentation in challenging weather conditions.

Abstract

We present IntrinsicWeather, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches. We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and 18k images under various weather conditions, each with intrinsic map annotations. IntrinsicWeather outperforms state-of-the-art pixel-space editing approaches, weather restoration methods, and rendering-based methods, showing promise for downstream tasks such as autonomous driving, enhancing the robustness of detection and segmentation in challenging weather scenarios.