Modular Neural Image Signal Processing

TL;DR

This paper introduces a modular neural image signal processing framework that enhances control, scalability, and generalization in image rendering, enabling diverse editing and style adaptation with high-quality results.

Contribution

It proposes a fully learning-based, modular neural ISP design that improves flexibility, scalability, and generalization over prior neural ISP methods.

Findings

Achieves high rendering accuracy and quality.

Supports diverse editing operations and styles.

Maintains moderate model size with competitive results.

Abstract

This paper presents a modular neural image signal processing (ISP) framework that processes raw inputs and renders high-quality display-referred images. Unlike prior neural ISP designs, our method introduces a high degree of modularity, providing full control over multiple intermediate stages of the rendering process.~This modular design not only achieves high rendering accuracy but also improves scalability, debuggability, generalization to unseen cameras, and flexibility to match different user-preference styles. To demonstrate the advantages of this design, we built a user-interactive photo-editing tool that leverages our neural ISP to support diverse editing operations and picture styles. The tool is carefully engineered to take advantage of the high-quality rendering of our neural ISP and to enable unlimited post-editable re-rendering. Our method is a fully learning-based framework with variants of different capacities, all of moderate size (ranging from ~0.5 M to ~3.9 M parameters for the entire pipeline), and consistently delivers competitive qualitative and quantitative results across multiple test sets. Watch the supplemental video at: https://youtu.be/ByhQjQSjxVM