Deep Contrastive Patch-Based Subspace Learning for Camera Image Signal Processing

TL;DR

This paper introduces a patch-based deep neural network model, PSL-AE, that enhances camera image processing by adaptively filtering heterogeneous artifacts using contrastive learning and subspace clustering.

Contribution

It proposes a novel patch-based, subspace learning autoencoder that handles non-uniform image distortions without assuming uniform artifact levels, improving robustness in image denoising.

Findings

Enhanced filtering of heterogeneous artifacts in images.

Effective in both synthetic and real-world noisy images.

Outperforms traditional uniform filtering methods.

Abstract

Camera Image Signal Processing (ISP) pipelines can get appealing results in different image signal processing tasks. Nonetheless, the majority of these methods, including those employing an encoder-decoder deep architecture for the task, typically utilize a uniform filter applied consistently across the entire image. However, it is natural to view a camera image as heterogeneous, as the color intensity and the artificial noise are distributed vastly differently, even across the two-dimensional domain of a single image. Varied Moire ringing, motion blur, color-bleaching, or lens-based projection distortions can all potentially lead to a heterogeneous image artifact filtering problem. In this paper, we present a specific patch-based, local subspace deep neural network that improves Camera ISP to be robust to heterogeneous artifacts (especially image denoising). We call our three-fold deep-trained model the Patch Subspace Learning Autoencoder (PSL-AE). The PSL-AE model does not make assumptions regarding uniform levels of image distortion. Instead, it first encodes patches extracted from noisy a nd clean image pairs, with different artifact types or distortion levels, by contrastive learning. Then, the patches of each image are encoded into corresponding soft clusters within their suitable latent sub-space, utilizing a prior mixture model. Furthermore, the decoders undergo training in an unsupervised manner, specifically trained for the image patches present in each cluster. The experiments highlight the adaptability and efficacy through enhanced heterogeneous filtering, both from synthesized artifacts but also realistic SIDD image pairs.