Controllable Weather Synthesis and Removal with Video Diffusion Models

TL;DR

WeatherWeaver is a novel video diffusion model that enables realistic, controllable weather effects synthesis and removal in videos, overcoming data scarcity and achieving superior results compared to existing methods.

Contribution

Introduces WeatherWeaver, a diffusion-based framework for controllable weather synthesis and removal in videos without requiring 3D models or paired data.

Findings

Outperforms state-of-the-art weather simulation methods

Produces high-quality, scene-preserving weather effects

Effective in real-world video applications

Abstract

Generating realistic and controllable weather effects in videos is valuable for many applications. Physics-based weather simulation requires precise reconstructions that are hard to scale to in-the-wild videos, while current video editing often lacks realism and control. In this work, we introduce WeatherWeaver, a video diffusion model that synthesizes diverse weather effects -- including rain, snow, fog, and clouds -- directly into any input video without the need for 3D modeling. Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability. To overcome the scarcity of paired training data, we propose a novel data strategy combining synthetic videos, generative image editing, and auto-labeled real-world videos. Extensive evaluations show that our method outperforms state-of-the-art methods in weather simulation and removal, providing high-quality, physically plausible, and scene-identity-preserving results over various real-world videos.