Experts-Guided Unbalanced Optimal Transport for ISP Learning from Unpaired and/or Paired Data

TL;DR

This paper introduces a novel unsupervised framework using Unbalanced Optimal Transport for training ISP pipelines with unpaired or paired data, improving robustness and performance over existing methods.

Contribution

It is the first to apply Unbalanced Optimal Transport to ISP learning, enabling effective training with unpaired data and introducing a committee of expert discriminators for targeted guidance.

Findings

Unpaired training achieves performance comparable to paired methods.

The framework outperforms original methods in paired mode across all metrics.

Robustness to outliers and specific ISP failure modes is improved.

Abstract

Learned Image Signal Processing (ISP) pipelines offer powerful end-to-end performance but are critically dependent on large-scale paired raw-to-sRGB datasets. This reliance on costly-to-acquire paired data remains a significant bottleneck. To address this challenge, we introduce a novel, unsupervised training framework based on Optimal Transport capable of training arbitrary ISP architectures in both unpaired and paired modes. We are the first to successfully apply Unbalanced Optimal Transport (UOT) for this complex, cross-domain translation task. Our UOT-based framework provides robustness to outliers in the target sRGB data, allowing it to discount atypical samples that would be prohibitively costly to map. A key component of our framework is a novel ``committee of expert discriminators,'' a hybrid adversarial regularizer. This committee guides the optimal transport mapping by providing specialized, targeted gradients to correct specific ISP failure modes, including color fidelity, structural artifacts, and frequency-domain realism. To demonstrate the superiority of our approach, we retrained existing state-of-the-art ISP architectures using our paired and unpaired setups. Our experiments show that while our framework, when trained in paired mode, exceeds the performance of the original paired methods across all metrics, our unpaired mode concurrently achieves quantitative and qualitative performance that rivals, and in some cases surpasses, the original paired-trained counterparts. The code and pre-trained models are available at: https://github.com/gosha20777/EGUOT-ISP.git.