Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data

TL;DR

This paper presents a fast, controllable shadow generation method using a single-step diffusion model trained on synthetic data, enabling real-time, high-quality shadows for 2D images with good real-world generalization.

Contribution

A novel synthetic-data training approach for controllable shadow generation with a single-step diffusion model, achieving real-time performance and generalization to real images.

Findings

Single-step diffusion achieves high-quality shadows with real-time speed.

Synthetic data training enables controllable shadow generation without 3D scene info.

Model generalizes well to real-world images, validated by experiments.

Abstract

Realistic shadow generation is a critical component for high-quality image compositing and visual effects, yet existing methods suffer from certain limitations: Physics-based approaches require a 3D scene geometry, which is often unavailable, while learning-based techniques struggle with control and visual artifacts. We introduce a novel method for fast, controllable, and background-free shadow generation for 2D object images. We create a large synthetic dataset using a 3D rendering engine to train a diffusion model for controllable shadow generation, generating shadow maps for diverse light source parameters. Through extensive ablation studies, we find that rectified flow objective achieves high-quality results with just a single sampling step enabling real-time applications. Furthermore, our experiments demonstrate that the model generalizes well to real-world images. To facilitate further research in evaluating quality and controllability in shadow generation, we release a new public benchmark containing a diverse set of object images and shadow maps in various settings. The project page is available at https://gojasper.github.io/controllable-shadow-generation-project/