Joint Demosaicing and Super-Resolution (JDSR): Network Design and Perceptual Optimization

TL;DR

This paper introduces a joint approach to image demosaicing and super-resolution using a novel neural network architecture, achieving superior image quality and perceptual results, with practical validation on real-world data.

Contribution

It proposes a new end-to-end network design for joint demosaicing and super-resolution, and incorporates perceptual optimization techniques to enhance visual quality.

Findings

RDSEN outperforms previous architectures like RCAN in PSNR/SSIM.

TRaGAN improves perceptual quality and reduces distortion in reconstructed images.

JDSR benefits high-quality image reconstruction from real-world Bayer data.

Abstract

Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this paper, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supported by a pre-demosaicing network (PDNet) as the pre-processing step. We address the issue of spatio-spectral attention for color-filter-array (CFA) data and discuss how to achieve better information flow by concatenating Residue-Dense Squeeze-and-Excitation Blocks (RDSEBs) for JDSR. Experimental results have shown that significant PSNR/SSIM gain can be achieved by RDSEN over previous network architectures including state-of-the-art RCAN. On perceptual optimization, we propose to leverage the latest ideas including relativistic discriminator and pre-excitation perceptual loss function to further improve the visual quality of textured regions in reconstructed images. Our extensive experiment results have shown that Texture-enhanced Relativistic average Generative Adversarial Network (TRaGAN) can produce both subjectively more pleasant images and objectively lower perceptual distortion scores than standard GAN for JDSR. Finally, we have verified the benefit of JDSR to high-quality image reconstruction from real-world Bayer pattern data collected by NASA Mars Curiosity.