Improved Mapping Between Illuminations and Sensors for RAW Images

TL;DR

This paper introduces a new dataset and a lightweight neural network method for mapping RAW images across different sensors and illuminations, facilitating better data augmentation and sensor interoperability in imaging tasks.

Contribution

The paper presents the first dataset capturing scenes under diverse illuminations and sensors, and proposes a novel neural network approach for illumination and sensor mapping.

Findings

Our method outperforms existing approaches in mapping accuracy.

The dataset enables robust training for sensor and illumination transfer.

Improves neural ISP training with cross-sensor illumination data.

Abstract

RAW images are unprocessed camera sensor output with sensor-specific RGB values based on the sensor's color filter spectral sensitivities. RAW images also incur strong color casts due to the sensor's response to the spectral properties of scene illumination. The sensor- and illumination-specific nature of RAW images makes it challenging to capture RAW datasets for deep learning methods, as scenes need to be captured for each sensor and under a wide range of illumination. Methods for illumination augmentation for a given sensor and the ability to map RAW images between sensors are important for reducing the burden of data capture. To explore this problem, we introduce the first-of-its-kind dataset comprising carefully captured scenes under a wide range of illumination. Specifically, we use a customized lightbox with tunable illumination spectra to capture several scenes with different cameras. Our illumination and sensor mapping dataset has 390 illuminations, four cameras, and 18 scenes. Using this dataset, we introduce a lightweight neural network approach for illumination and sensor mapping that outperforms competing methods. We demonstrate the utility of our approach on the downstream task of training a neural ISP. Link to project page: https://github.com/SamsungLabs/illum-sensor-mapping.