To See in the Dark: N2DGAN for Background Modeling in Nighttime Scene

TL;DR

This paper introduces N2DGAN, a novel GAN-based framework that transforms nighttime video frames into virtual daytime images to improve background modeling for nighttime surveillance, addressing challenges of low contrast and noise.

Contribution

The paper proposes a new generation-based background modeling method using N2DGAN, combining global and local networks with spatial-temporal constraints for nighttime scene analysis.

Findings

Effective nighttime-to-daytime image translation improves background modeling accuracy.

The multi-scale Bayes model captures background temporal variations.

Results outperform existing methods on collected datasets.

Abstract

Due to the deteriorated conditions of \mbox{illumination} lack and uneven lighting, nighttime images have lower contrast and higher noise than their daytime counterparts of the same scene, which limits seriously the performances of conventional background modeling methods. For such a challenging problem of background modeling under nighttime scene, an innovative and reasonable solution is proposed in this paper, which paves a new way completely different from the existing ones. To make background modeling under nighttime scene performs as well as in daytime condition, we put forward a promising generation-based background modeling framework for foreground surveillance. With a pre-specified daytime reference image as background frame, the {\bfseries GAN} based generation model, called {\bfseries N2DGAN}, is trained to transfer each frame of {\bfseries n}ighttime video {\bfseries to} a virtual {\bfseries d}aytime image with the same scene to the reference image except for the foreground region. Specifically, to balance the preservation of background scene and the foreground object(s) in generating the virtual daytime image, we present a two-pathway generation model, in which the global and local sub-networks are well combined with spatial and temporal consistency constraints. For the sequence of generated virtual daytime images, a multi-scale Bayes model is further proposed to characterize pertinently the temporal variation of background. We evaluate on collected datasets with manually labeled ground truth, which provides a valuable resource for related research community. The impressive results illustrated in both the main paper and supplementary show efficacy of our proposed approach.