SoftShadow: Leveraging Soft Masks for Penumbra-Aware Shadow Removal

TL;DR

SoftShadow introduces soft masks based on physical shadow models and pretrained SAM to improve shadow removal, especially near boundaries, achieving state-of-the-art results and better generalization.

Contribution

The paper proposes a novel soft mask generation method using physical constraints and pretrained models, enhancing shadow removal accuracy and boundary artifact reduction.

Findings

Achieves state-of-the-art shadow removal performance.

Better boundary artifact restoration compared to binary masks.

Demonstrates superior generalizability across datasets.

Abstract

Recent advancements in deep learning have yielded promising results for the image shadow removal task. However, most existing methods rely on binary pre-generated shadow masks. The binary nature of such masks could potentially lead to artifacts near the boundary between shadow and non-shadow areas. In view of this, inspired by the physical model of shadow formation, we introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a SoftShadow framework by leveraging the prior knowledge of pretrained SAM and integrating physical constraints. Specifically, we jointly tune the SAM and the subsequent shadow removal network using penumbra formation constraint loss, mask reconstruction loss, and shadow removal loss. This framework enables accurate predictions of penumbra (partially shaded) and umbra (fully shaded) areas while simultaneously facilitating end-to-end shadow removal. Through extensive experiments on popular datasets, we found that our SoftShadow framework, which generates soft masks, can better restore boundary artifacts, achieve state-of-the-art performance, and demonstrate superior generalizability.