Uni-ISP: Toward Unifying the Learning of ISPs from Multiple Mobile Cameras

TL;DR

Uni-ISP introduces a unified learning framework for diverse mobile camera ISPs, leveraging device-aware embeddings to enhance accuracy, versatility, and adaptability across different camera models, supported by a new real-world dataset.

Contribution

The paper presents Uni-ISP, a novel unified ISP learning pipeline that effectively handles multiple mobile cameras using device-aware embeddings and a specialized training scheme.

Findings

Improves forward and inverse ISP performance (+2.4dB/1.5dB PSNR).

Demonstrates high accuracy and adaptability across diverse camera models.

Enables new applications previously inaccessible to conventional ISPs.

Abstract

Modern end-to-end image signal processors (ISPs) can learn complex mappings from RAW/XYZ data to sRGB (and vice versa), opening new possibilities in image processing. However, the growing diversity of camera models, particularly in mobile devices, renders the development of individual ISPs unsustainable due to their limited versatility and adaptability across varied camera systems. In this paper, we introduce Uni-ISP, a novel pipeline that unifies ISP learning for diverse mobile cameras, delivering a highly accurate and adaptable processor. The core of Uni-ISP is leveraging device-aware embeddings through learning forward/inverse ISPs and its special training scheme. By doing so, Uni-ISP not only improves the performance of forward and inverse ISPs but also unlocks new applications previously inaccessible to conventional learned ISPs. To support this work, we construct a real-world 4K dataset, FiveCam, comprising more than 2,400 pairs of sRGB-RAW images captured synchronously by five smartphone cameras. Extensive experiments validate Uni-ISP's accuracy in learning forward and inverse ISPs (with improvements of +2.4dB/1.5dB PSNR), versatility in enabling new applications, and adaptability to new camera models.