Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal

TL;DR

This paper introduces a novel multi-task framework using stacked conditional GANs to jointly detect and remove shadows from images, improving performance through mutual benefits and global scene understanding.

Contribution

It proposes the first end-to-end multi-task shadow detection and removal framework with a novel stacked CGAN architecture that enhances both tasks mutually.

Findings

Outperforms state-of-the-art methods on multiple datasets.

Constructed the first large-scale shadow dataset with 1870 triplets.

Demonstrates significant improvements in shadow detection and removal accuracy.

Abstract

Understanding shadows from a single image spontaneously derives into two types of task in previous studies, containing shadow detection and shadow removal. In this paper, we present a multi-task perspective, which is not embraced by any existing work, to jointly learn both detection and removal in an end-to-end fashion that aims at enjoying the mutually improved benefits from each other. Our framework is based on a novel STacked Conditional Generative Adversarial Network (ST-CGAN), which is composed of two stacked CGANs, each with a generator and a discriminator. Specifically, a shadow image is fed into the first generator which produces a shadow detection mask. That shadow image, concatenated with its predicted mask, goes through the second generator in order to recover its shadow-free image consequently. In addition, the two corresponding discriminators are very likely to model higher level relationships and global scene characteristics for the detected shadow region and reconstruction via removing shadows, respectively. More importantly, for multi-task learning, our design of stacked paradigm provides a novel view which is notably different from the commonly used one as the multi-branch version. To fully evaluate the performance of our proposed framework, we construct the first large-scale benchmark with 1870 image triplets (shadow image, shadow mask image, and shadow-free image) under 135 scenes. Extensive experimental results consistently show the advantages of ST-CGAN over several representative state-of-the-art methods on two large-scale publicly available datasets and our newly released one.