Recursive Frequency Selective Reconstruction of Non-Regularly Sampled Video Data

TL;DR

This paper introduces a recursive multi-frame reconstruction method for non-regularly sampled video data, significantly improving image quality and enabling real-time processing by leveraging temporal correlations and a novel reference order.

Contribution

It proposes a new recursive multi-frame approach with a novel reference order and weighting function, enhancing reconstruction quality over existing methods.

Findings

Achieves up to 1.13 dB PSNR gain over single-frame methods

Outperforms existing multi-frame approaches with 0.31 dB PSNR improvement

Capable of real-time processing without pre-reconstruction steps

Abstract

High resolution images can be acquired using a non-regular sampling sensor which consists of an underlying low resolution sensor that is covered with a non-regular sampling mask. The reconstructed high resolution image is then obtained during post-processing. Recently, it has been shown that the temporal correlation between neighboring frames can be exploited in order to enhance the reconstruction quality of non-regularly sampled video data. In this paper, a new recursive multi-frame reconstruction approach is proposed in order to further increase the reconstruction quality. By using a new reference order, previously reconstructed frames can be used for the subsequent motion estimation and a new weighting function allows for the incorporation of multiple pixels projected onto the same position. With the new recursive multi-frame approach, a visually noticeable average gain in PSNR of up to 1.13 dB with respect to a state-of-the-art single-frame reconstruction approach can be achieved. Compared to the existing multi-frame approach, a gain of 0.31 dB is possible. SSIM results show the same behavior as PSNR results. Additionally, the pre-reconstruction step of the existing multi-frame approach can be avoided and the new algorithm is, in general, capable of real-time processing.