Model-Based Image Signal Processors via Learnable Dictionaries

TL;DR

This paper introduces a hybrid, learnable, and interpretable image signal processor (ISP) model that uses dictionaries for bidirectional RAW-RGB mapping, enabling high-quality image reconstruction and denoising with limited data.

Contribution

The authors propose a novel invertible, dictionary-based ISP model that combines interpretability with end-to-end learnability, addressing data scarcity and control issues in traditional methods.

Findings

Achieves state-of-the-art results in RAW image reconstruction.

Performs effectively in RAW denoising tasks with limited data.

Enables data augmentation through learned dictionaries.

Abstract

Digital cameras transform sensor RAW readings into RGB images by means of their Image Signal Processor (ISP). Computational photography tasks such as image denoising and colour constancy are commonly performed in the RAW domain, in part due to the inherent hardware design, but also due to the appealing simplicity of noise statistics that result from the direct sensor readings. Despite this, the availability of RAW images is limited in comparison with the abundance and diversity of available RGB data. Recent approaches have attempted to bridge this gap by estimating the RGB to RAW mapping: handcrafted model-based methods that are interpretable and controllable usually require manual parameter fine-tuning, while end-to-end learnable neural networks require large amounts of training data, at times with complex training procedures, and generally lack interpretability and parametric control. Towards addressing these existing limitations, we present a novel hybrid model-based and data-driven ISP that builds on canonical ISP operations and is both learnable and interpretable. Our proposed invertible model, capable of bidirectional mapping between RAW and RGB domains, employs end-to-end learning of rich parameter representations, i.e. dictionaries, that are free from direct parametric supervision and additionally enable simple and plausible data augmentation. We evidence the value of our data generation process by extensive experiments under both RAW image reconstruction and RAW image denoising tasks, obtaining state-of-the-art performance in both. Additionally, we show that our ISP can learn meaningful mappings from few data samples, and that denoising models trained with our dictionary-based data augmentation are competitive despite having only few or zero ground-truth labels.