Flow-Guided Sparse Transformer for Video Deblurring

TL;DR

This paper introduces a Flow-Guided Sparse Transformer framework for video deblurring that effectively captures long-range dependencies and non-local self-similarity, outperforming existing methods on standard datasets.

Contribution

The novel FGST framework with a flow-guided sparse self-attention module and recurrent embedding mechanism enhances long-range temporal modeling in video deblurring.

Findings

Outperforms state-of-the-art methods on DVD and GOPRO datasets.

Produces more visually pleasing results in real video deblurring.

Effectively captures long-range dependencies and non-local self-similarity.

Abstract

Exploiting similar and sharper scene patches in spatio-temporal neighborhoods is critical for video deblurring. However, CNN-based methods show limitations in capturing long-range dependencies and modeling non-local self-similarity. In this paper, we propose a novel framework, Flow-Guided Sparse Transformer (FGST), for video deblurring. In FGST, we customize a self-attention module, Flow-Guided Sparse Window-based Multi-head Self-Attention (FGSW-MSA). For each $query$ element on the blurry reference frame, FGSW-MSA enjoys the guidance of the estimated optical flow to globally sample spatially sparse yet highly related $key$ elements corresponding to the same scene patch in neighboring frames. Besides, we present a Recurrent Embedding (RE) mechanism to transfer information from past frames and strengthen long-range temporal dependencies. Comprehensive experiments demonstrate that our proposed FGST outperforms state-of-the-art (SOTA) methods on both DVD and GOPRO datasets and even yields more visually pleasing results in real video deblurring. Code and pre-trained models are publicly available at https://github.com/linjing7/VR-Baseline