Deformable Kernel Convolutional Network for Video Extreme Super-Resolution

TL;DR

This paper introduces DKSAN, a novel deep learning model that effectively exploits spatial and temporal redundancies for video super-resolution, especially at large scale factors, outperforming existing methods.

Contribution

The paper proposes DKSAN with new DKC_Align and DKSA modules to better utilize spatial and temporal information in video super-resolution tasks.

Findings

Outperforms EDVR on Vid3oC and IntVID datasets

Achieves superior subjective and objective quality

Effective at super-resolving videos with scale factor up to 16

Abstract

Video super-resolution, which attempts to reconstruct high-resolution video frames from their corresponding low-resolution versions, has received increasingly more attention in recent years. Most existing approaches opt to use deformable convolution to temporally align neighboring frames and apply traditional spatial attention mechanism (convolution based) to enhance reconstructed features. However, such spatial-only strategies cannot fully utilize temporal dependency among video frames. In this paper, we propose a novel deep learning based VSR algorithm, named Deformable Kernel Spatial Attention Network (DKSAN). Thanks to newly designed Deformable Kernel Convolution Alignment (DKC_Align) and Deformable Kernel Spatial Attention (DKSA) modules, DKSAN can better exploit both spatial and temporal redundancies to facilitate the information propagation across different layers. We have tested DKSAN on AIM2020 Video Extreme Super-Resolution Challenge to super-resolve videos with a scale factor as large as 16. Experimental results demonstrate that our proposed DKSAN can achieve both better subjective and objective performance compared with the existing state-of-the-art EDVR on Vid3oC and IntVID datasets.