Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularization

TL;DR

This paper introduces SemiVDN, a semi-supervised video desnowing network that leverages unlabeled real snowy videos and a novel contrastive regularization to improve snow removal in real-world scenarios, outperforming existing methods.

Contribution

The paper proposes a semi-supervised framework with a distribution-driven contrastive regularization and a prior-guided temporal decoupling module for effective real-world video desnowing.

Findings

SemiVDN outperforms state-of-the-art methods on benchmark and real snowy videos.

Contrastive regularization reduces distribution gap between synthetic and real data.

The proposed model effectively preserves background details while removing snow.

Abstract

Snow degradations present formidable challenges to the advancement of computer vision tasks by the undesirable corruption in outdoor scenarios. While current deep learning-based desnowing approaches achieve success on synthetic benchmark datasets, they struggle to restore out-of-distribution real-world snowy videos due to the deficiency of paired real-world training data. To address this bottleneck, we devise a new paradigm for video desnowing in a semi-supervised spirit to involve unlabeled real data for the generalizable snow removal. Specifically, we construct a real-world dataset with 85 snowy videos, and then present a Semi-supervised Video Desnowing Network (SemiVDN) equipped by a novel Distribution-driven Contrastive Regularization. The elaborated contrastive regularization mitigates the distribution gap between the synthetic and real data, and consequently maintains the desired snow-invariant background details. Furthermore, based on the atmospheric scattering model, we introduce a Prior-guided Temporal Decoupling Experts module to decompose the physical components that make up a snowy video in a frame-correlated manner. We evaluate our SemiVDN on benchmark datasets and the collected real snowy data. The experimental results demonstrate the superiority of our approach against state-of-the-art image- and video-level desnowing methods.