Physics-based Shadow Image Decomposition for Shadow Removal

TL;DR

This paper introduces a physics-inspired deep learning framework for shadow removal that decomposes images into shadow-free components, improves accuracy, and enables training without shadow-free images, validated on challenging datasets.

Contribution

The paper presents a novel physics-based shadow decomposition model combined with deep networks, enabling shadow removal without needing shadow-free training data.

Findings

20% RMSE improvement over state-of-the-art

Effective shadow removal without shadow-free images

New dataset for video shadow removal

Abstract

We propose a novel deep learning method for shadow removal. Inspired by physical models of shadow formation, we use a linear illumination transformation to model the shadow effects in the image that allows the shadow image to be expressed as a combination of the shadow-free image, the shadow parameters, and a matte layer. We use two deep networks, namely SP-Net and M-Net, to predict the shadow parameters and the shadow matte respectively. This system allows us to remove the shadow effects from images. We then employ an inpainting network, I-Net, to further refine the results. We train and test our framework on the most challenging shadow removal dataset (ISTD). Our method improves the state-of-the-art in terms of root mean square error (RMSE) for the shadow area by 20\%. Furthermore, this decomposition allows us to formulate a patch-based weakly-supervised shadow removal method. This model can be trained without any shadow-free images (that are cumbersome to acquire) and achieves competitive shadow removal results compared to state-of-the-art methods that are trained with fully paired shadow and shadow-free images. Last, we introduce SBU-Timelapse, a video shadow removal dataset for evaluating shadow removal methods.