WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation

TL;DR

WeatherCity is a novel framework for 4D urban scene reconstruction that enables controllable, high-fidelity weather editing and dynamic weather simulation, improving over existing methods in flexibility and realism.

Contribution

It introduces a weather Gaussian representation with scene features and weather-specific decoders, combined with a physics-driven model for dynamic weather effects, enabling flexible weather editing and scene reconstruction.

Findings

Achieves high fidelity and temporal consistency in 4D scenes.

Supports fine-grained weather control and object-level manipulation.

Demonstrates effectiveness across multiple datasets and scenes.

Abstract

Editable high-fidelity 4D scenes are crucial for autonomous driving, as they can be applied to end-to-end training and closed-loop simulation. However, existing reconstruction methods are primarily limited to replicating observed scenes and lack the capability for diverse weather simulation. While image-level weather editing methods tend to introduce scene artifacts and offer poor controllability over the weather effects. To address these limitations, we propose WeatherCity, a novel framework for 4D urban scene reconstruction and weather editing. Specifically, we leverage a text-guided image editing model to achieve flexible editing of image weather backgrounds. To tackle the challenge of multi-weather modeling, we introduce a novel weather Gaussian representation based on shared scene features and dedicated weather-specific decoders. This representation is further enhanced with a content consistency optimization, ensuring coherent modeling across different weather conditions. Additionally, we design a physics-driven model that simulates dynamic weather effects through particles and motion patterns. Extensive experiments on multiple datasets and various scenes demonstrate that WeatherCity achieves flexible controllability, high fidelity, and temporal consistency in 4D reconstruction and weather editing. Our framework not only enables fine-grained control over weather conditions (e.g., light rain and heavy snow) but also supports object-level manipulation within the scene.