3DSRnet: Video Super-resolution using 3D Convolutional Neural Networks

TL;DR

This paper introduces 3DSRnet, a 3D convolutional neural network for video super-resolution that effectively captures spatio-temporal features without motion alignment, outperforming previous methods on benchmark datasets.

Contribution

The paper presents a novel 3D-CNN architecture for video super-resolution that preserves temporal information and handles scene changes, without requiring motion alignment preprocessing.

Findings

Outperforms state-of-the-art methods by 0.45 and 0.36 dB in PSNR for scales 3 and 4.

Effectively captures nonlinear temporal characteristics in videos.

Addresses scene change issues in video super-resolution.

Abstract

In video super-resolution, the spatio-temporal coherence between, and among the frames must be exploited appropriately for accurate prediction of the high resolution frames. Although 2D convolutional neural networks (CNNs) are powerful in modelling images, 3D-CNNs are more suitable for spatio-temporal feature extraction as they can preserve temporal information. To this end, we propose an effective 3D-CNN for video super-resolution, called the 3DSRnet that does not require motion alignment as preprocessing. Our 3DSRnet maintains the temporal depth of spatio-temporal feature maps to maximally capture the temporally nonlinear characteristics between low and high resolution frames, and adopts residual learning in conjunction with the sub-pixel outputs. It outperforms the most state-of-the-art method with average 0.45 and 0.36 dB higher in PSNR for scales 3 and 4, respectively, in the Vidset4 benchmark. Our 3DSRnet first deals with the performance drop due to scene change, which is important in practice but has not been previously considered.