Shadow Removal Refinement via Material-Consistent Shadow Edges

TL;DR

This paper introduces a novel shadow removal refinement method that leverages material-consistent shadow edges and self-supervision, improving results on complex in-the-wild images and providing new evaluation metrics and datasets.

Contribution

It proposes a new approach to identify material-consistent shadow edges using a fine-tuned segmentation model for self-supervision in shadow removal.

Findings

Outperforms state-of-the-art shadow removal methods on challenging images.

Introduces a new metric and dataset for evaluating shadow removal without paired data.

Enhances shadow removal quality by enforcing color and texture consistency.

Abstract

Shadow boundaries can be confused with material boundaries as both exhibit sharp changes in luminance or contrast within a scene. However, shadows do not modify the intrinsic color or texture of surfaces. Therefore, on both sides of shadow edges traversing regions with the same material, the original color and textures should be the same if the shadow is removed properly. These shadow/shadow-free pairs are very useful but hard-to-collect supervision signals. The crucial contribution of this paper is to learn how to identify those shadow edges that traverse material-consistent regions and how to use them as self-supervision for shadow removal refinement during test time. To achieve this, we fine-tune SAM, an image segmentation foundation model, to produce a shadow-invariant segmentation and then extract material-consistent shadow edges by comparing the SAM segmentation with the shadow mask. Utilizing these shadow edges, we introduce color and texture-consistency losses to enhance the shadow removal process. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging, in-the-wild images, outperforming the state-of-the-art shadow removal methods. Additionally, we propose a new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data.