IntrinsicWeather: Controllable Weather Editing in Intrinsic Space
Yixin Zhu, Zuo-Liang Zhu, Jian Yang, Milo\v{s} Ha\v{s}an, Jin Xie, Beibei Wang

TL;DR
IntrinsicWeather is a diffusion-based framework enabling controllable weather editing in images through intrinsic maps and text prompts, improving spatial accuracy and robustness in outdoor scene editing.
Contribution
It introduces a novel intrinsic map-aware diffusion framework with a new attention mechanism and a large annotated dataset for weather editing in outdoor scenes.
Findings
Outperforms state-of-the-art pixel-space editing methods.
Enables fine-grained weather control via CLIP-space interpolation.
Enhances robustness of detection and segmentation in challenging weather conditions.
Abstract
We present IntrinsicWeather, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches. We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
