Learning Shadow Correspondence for Video Shadow Detection

TL;DR

This paper introduces SC-Cor, a weakly-supervised, plug-and-play module that improves temporal consistency in video shadow detection by enhancing shadow feature correspondence across frames, outperforming previous methods.

Contribution

The paper proposes a novel SC-Cor method that learns pixel-wise shadow correspondence across frames without dense labels, improving temporal stability and robustness in video shadow detection.

Findings

SC-Cor outperforms previous state-of-the-art by 6.51% IoU.

SC-Cor improves temporal stability of shadow detection.

The method is computationally efficient and easy to integrate.

Abstract

Video shadow detection aims to generate consistent shadow predictions among video frames. However, the current approaches suffer from inconsistent shadow predictions across frames, especially when the illumination and background textures change in a video. We make an observation that the inconsistent predictions are caused by the shadow feature inconsistency, i.e., the features of the same shadow regions show dissimilar proprieties among the nearby frames.In this paper, we present a novel Shadow-Consistent Correspondence method (SC-Cor) to enhance pixel-wise similarity of the specific shadow regions across frames for video shadow detection. Our proposed SC-Cor has three main advantages. Firstly, without requiring the dense pixel-to-pixel correspondence labels, SC-Cor can learn the pixel-wise correspondence across frames in a weakly-supervised manner. Secondly, SC-Cor considers intra-shadow separability, which is robust to the variant textures and illuminations in videos. Finally, SC-Cor is a plug-and-play module that can be easily integrated into existing shadow detectors with no extra computational cost. We further design a new evaluation metric to evaluate the temporal stability of the video shadow detection results. Experimental results show that SC-Cor outperforms the prior state-of-the-art method, by 6.51% on IoU and 3.35% on the newly introduced temporal stability metric.