Raw Bayer Pattern Image Synthesis for Computer Vision-oriented Image Signal Processing Pipeline Design

TL;DR

This paper introduces a novel GAN-based method for synthesizing high-quality RAW Bayer images of arbitrary sizes, improving dataset availability for ISP pipeline research and enabling end-to-end vision task training.

Contribution

The proposed method leverages data transformations in GAN training to generate realistic RAW Bayer images, addressing dataset scarcity and enhancing training stability.

Findings

Generated images outperform existing methods in FID, PSNR, and MSSIM.

Training stability is improved with the proposed approach.

Synthesized RAW images enable effective end-to-end vision task training.

Abstract

In this paper, we propose a method to add constraints that are un-formulatable in generative adversarial networks (GAN)-based arbitrary size RAW Bayer image generation. It is shown theoretically that by using the transformed data in GAN training, it is able to improve the learning of the original data distribution, owing to the invariant of Jensen-Shannon (JS) divergence between two distributions under invertible and differentiable transformation. Benefiting from the proposed method, RAW Bayer pattern images can be generated by configuring the transformation as demosaicing. It is shown that by adding another transformation, the proposed method is able to synthesize high-quality RAW Bayer images with arbitrary size. Experimental results show that images generated by the proposed method outperform the existing methods in the Fr\'echet inception distance (FID) score, peak signal to noise ratio (PSNR), and mean structural similarity (MSSIM), and the training process is more stable. To the best knowledge of the authors, there is no open-source, large-scale image dataset in the RAW Bayer domain, which is crucial for research works aiming to explore the image signal processing (ISP) pipeline design for computer vision tasks. Converting the existing commonly used color image datasets to their corresponding RAW Bayer versions, the proposed method can be a promising solution to the RAW image dataset problem. We also show in the experiments that, by training object detection frameworks using the synthesized RAW Bayer images, they can be used in an end-to-end manner (from RAW images to vision tasks) with negligible performance degradation.