Spatio-Temporal Filter Adaptive Network for Video Deblurring

TL;DR

This paper introduces a unified spatio-temporal filter adaptive network for video deblurring that dynamically generates spatially adaptive filters to improve alignment and deblurring, outperforming existing methods.

Contribution

The proposed STFAN integrates alignment and deblurring into a single framework using a novel FAC layer for adaptive filtering, reducing artifacts and improving performance.

Findings

Outperforms state-of-the-art methods in accuracy

Achieves faster processing with smaller model size

Effectively handles spatially variant blur in real-world videos

Abstract

Video deblurring is a challenging task due to the spatially variant blur caused by camera shake, object motions, and depth variations, etc. Existing methods usually estimate optical flow in the blurry video to align consecutive frames or approximate blur kernels. However, they tend to generate artifacts or cannot effectively remove blur when the estimated optical flow is not accurate. To overcome the limitation of separate optical flow estimation, we propose a Spatio-Temporal Filter Adaptive Network (STFAN) for the alignment and deblurring in a unified framework. The proposed STFAN takes both blurry and restored images of the previous frame as well as blurry image of the current frame as input, and dynamically generates the spatially adaptive filters for the alignment and deblurring. We then propose the new Filter Adaptive Convolutional (FAC) layer to align the deblurred features of the previous frame with the current frame and remove the spatially variant blur from the features of the current frame. Finally, we develop a reconstruction network which takes the fusion of two transformed features to restore the clear frames. Both quantitative and qualitative evaluation results on the benchmark datasets and real-world videos demonstrate that the proposed algorithm performs favorably against state-of-the-art methods in terms of accuracy, speed as well as model size.