Day-to-Night Image Synthesis for Training Nighttime Neural ISPs

TL;DR

This paper introduces a method to synthesize realistic nighttime images from daytime images to facilitate training neural image signal processors for night mode, reducing the need for tedious real data collection.

Contribution

The authors propose a novel framework for converting daytime raw images into realistic nighttime images, enabling easier generation of paired datasets for training nightmode neural ISPs.

Findings

Synthetic nighttime images improve ISP training performance.

Combining synthetic and small real datasets yields near real-data training results.

The method reduces data collection effort for nighttime ISP datasets.

Abstract

Many flagship smartphone cameras now use a dedicated neural image signal processor (ISP) to render noisy raw sensor images to the final processed output. Training nightmode ISP networks relies on large-scale datasets of image pairs with: (1) a noisy raw image captured with a short exposure and a high ISO gain; and (2) a ground truth low-noise raw image captured with a long exposure and low ISO that has been rendered through the ISP. Capturing such image pairs is tedious and time-consuming, requiring careful setup to ensure alignment between the image pairs. In addition, ground truth images are often prone to motion blur due to the long exposure. To address this problem, we propose a method that synthesizes nighttime images from daytime images. Daytime images are easy to capture, exhibit low-noise (even on smartphone cameras) and rarely suffer from motion blur. We outline a processing framework to convert daytime raw images to have the appearance of realistic nighttime raw images with different levels of noise. Our procedure allows us to easily produce aligned noisy and clean nighttime image pairs. We show the effectiveness of our synthesis framework by training neural ISPs for nightmode rendering. Furthermore, we demonstrate that using our synthetic nighttime images together with small amounts of real data (e.g., 5% to 10%) yields performance almost on par with training exclusively on real nighttime images. Our dataset and code are available at https://github.com/SamsungLabs/day-to-night.