CRISPnet: Color Rendition ISP Net

TL;DR

CRISPnet is a novel deep learning-based image signal processing model that significantly improves color reproduction accuracy in smartphone images by mimicking legacy ISP heuristics and utilizing image metadata and scene semantics.

Contribution

It introduces CRISPnet, the first learned ISP model focused on color rendition, and provides a new dataset with diverse real-world and HDR data captured from actual smartphone ISPs.

Findings

CRISPnet outperforms existing models in color accuracy.

The model effectively uses image metadata and scene semantics.

New dataset enables better training and evaluation of learned ISPs.

Abstract

Image signal processors (ISPs) are historically grown legacy software systems for reconstructing color images from noisy raw sensor measurements. They are usually composited of many heuristic blocks for denoising, demosaicking, and color restoration. Color reproduction in this context is of particular importance, since the raw colors are often severely distorted, and each smart phone manufacturer has developed their own characteristic heuristics for improving the color rendition, for example of skin tones and other visually important colors. In recent years there has been strong interest in replacing the historically grown ISP systems with deep learned pipelines. Much progress has been made in approximating legacy ISPs with such learned models. However, so far the focus of these efforts has been on reproducing the structural features of the images, with less attention paid to color rendition. Here we present CRISPnet, the first learned ISP model to specifically target color rendition accuracy relative to a complex, legacy smart phone ISP. We achieve this by utilizing both image metadata (like a legacy ISP would), as well as by learning simple global semantics based on image classification -- similar to what a legacy ISP does to determine the scene type. We also contribute a new ISP image dataset consisting of both high dynamic range monitor data, as well as real-world data, both captured with an actual cell phone ISP pipeline under a variety of lighting conditions, exposure times, and gain settings.