A New Multi-Picture Architecture for Learned Video Deinterlacing and Demosaicing with Parallel Deformable Convolution and Self-Attention Blocks

TL;DR

This paper introduces a novel multi-picture architecture utilizing deformable convolution and self-attention for improved learned video deinterlacing and demosaicing, outperforming existing methods in quality metrics.

Contribution

The paper presents a new multi-picture architecture with modified deformable convolution and a residual efficient top-k self-attention block for better video deinterlacing and demosaicing.

Findings

Significantly exceeds state-of-the-art in PSNR and SSIM.

Effective in both synthetic and real-world datasets.

Ablation studies validate each component's benefit.

Abstract

Despite the fact real-world video deinterlacing and demosaicing are well-suited to supervised learning from synthetically degraded data because the degradation models are known and fixed, learned video deinterlacing and demosaicing have received much less attention compared to denoising and super-resolution tasks. We propose a new multi-picture architecture for video deinterlacing or demosaicing by aligning multiple supporting pictures with missing data to a reference picture to be reconstructed, benefiting from both local and global spatio-temporal correlations in the feature space using modified deformable convolution blocks and a novel residual efficient top-$k$ self-attention (kSA) block, respectively. Separate reconstruction blocks are used to estimate different types of missing data. Our extensive experimental results, on synthetic or real-world datasets, demonstrate that the proposed novel architecture provides superior results that significantly exceed the state-of-the-art for both tasks in terms of PSNR, SSIM, and perceptual quality. Ablation studies are provided to justify and show the benefit of each novel modification made to the deformable convolution and residual efficient kSA blocks. Code is available: https://github.com/KUIS-AI-Tekalp-Research-Group/Video-Deinterlacing.