Semantic-guided Adversarial Diffusion Model for Self-supervised Shadow Removal

TL;DR

This paper introduces a two-stage, self-supervised shadow removal framework combining semantic-guided GANs and diffusion models, improving image quality and semantic accuracy without relying on paired data.

Contribution

It proposes a novel semantic-guided adversarial diffusion approach with a multi-modal semantic prompter for enhanced self-supervised shadow removal.

Findings

Effective shadow removal demonstrated on multiple datasets.

Improved image texture and edge quality.

Better semantic consistency in restored images.

Abstract

Existing unsupervised methods have addressed the challenges of inconsistent paired data and tedious acquisition of ground-truth labels in shadow removal tasks. However, GAN-based training often faces issues such as mode collapse and unstable optimization. Furthermore, due to the complex mapping between shadow and shadow-free domains, merely relying on adversarial learning is not enough to capture the underlying relationship between two domains, resulting in low quality of the generated images. To address these problems, we propose a semantic-guided adversarial diffusion framework for self-supervised shadow removal, which consists of two stages. At first stage a semantic-guided generative adversarial network (SG-GAN) is proposed to carry out a coarse result and construct paired synthetic data through a cycle-consistent structure. Then the coarse result is refined with a diffusion-based restoration module (DBRM) to enhance the texture details and edge artifact at second stage. Meanwhile, we propose a multi-modal semantic prompter (MSP) that aids in extracting accurate semantic information from real images and text, guiding the shadow removal network to restore images better in SG-GAN. We conduct experiments on multiple public datasets, and the experimental results demonstrate the effectiveness of our method.