Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal

TL;DR

This paper introduces a novel diffusion-based portrait shadow removal method that leverages structure-guided inpainting and detail restoration to produce high-fidelity, artifact-free results, outperforming existing techniques.

Contribution

It proposes a structure-guided diffusion framework with separate inpainting and detail restoration models for improved portrait shadow removal.

Findings

Outperforms existing shadow removal methods on benchmark datasets.

Effectively preserves facial identity and fine details.

Reduces common artifacts like shadow residuals and color distortions.

Abstract

We present a diffusion-based portrait shadow removal approach that can robustly produce high-fidelity results. Unlike previous methods, we cast shadow removal as diffusion-based inpainting. To this end, we first train a shadow-independent structure extraction network on a real-world portrait dataset with various synthetic lighting conditions, which allows to generate a shadow-independent structure map including facial details while excluding the unwanted shadow boundaries. The structure map is then used as condition to train a structure-guided inpainting diffusion model for removing shadows in a generative manner. Finally, to restore the fine-scale details (e.g., eyelashes, moles and spots) that may not be captured by the structure map, we take the gradients inside the shadow regions as guidance and train a detail restoration diffusion model to refine the shadow removal result. Extensive experiments on the benchmark datasets show that our method clearly outperforms existing methods, and is effective to avoid previously common issues such as facial identity tampering, shadow residual, color distortion, structure blurring, and loss of details. Our code is available at https://github.com/wanchang-yu/Structure-Guided-Diffusion-for-Portrait-Shadow-Removal.