Joint Demosaicking / Rectification of Fisheye Camera Images using Multi-color Graph Laplacian Regularization

TL;DR

This paper introduces a joint demosaicking and rectification method for fisheye camera images using graph Laplacian regularization, reducing interpolation errors and noise pollution compared to sequential processing.

Contribution

The proposed framework performs demosaicking and grid mapping simultaneously, improving image quality by limiting noise accumulation and interpolation errors.

Findings

Outperforms sequential methods in PSNR and SSIM metrics.

Achieves up to 0.52 dB PSNR improvement on public dataset.

Achieves up to 5.53 dB PSNR improvement on in-house dataset.

Abstract

To compose a 360 image from a rig with multiple fisheye cameras, a conventional processing pipeline first performs demosaicking on each fisheye camera's Bayer-patterned grid, then translates demosaicked pixels from the camera grid to a rectified image grid---thus performing two image interpolation steps in sequence. Hence interpolation errors can accumulate, and acquisition noise in the captured pixels can pollute neighbors in two consecutive processing stages. In this paper, we propose a joint processing framework that performs demosaicking and grid-to-grid mapping simultaneously---thus limiting noise pollution to one interpolation. Specifically, we first obtain a reverse mapping function from a regular on-grid location in the rectified image to an irregular off-grid location in the camera's Bayer-patterned image. For each pair of adjacent pixels in the rectified grid, we estimate its gradient using the pair's neighboring pixel gradients in three colors in the Bayer-patterned grid. We construct a similarity graph based on the estimated gradients, and interpolate pixels in the rectified grid directly via graph Laplacian regularization (GLR). Experiments show that our joint method outperforms several competing local methods that execute demosaicking and rectification in sequence, by up to 0.52 dB in PSNR and 0.086 in SSIM on the publicly available dataset, and by up to 5.53dB in PSNR and 0.411 in SSIM on the in-house constructed dataset.