VDTR: Video Deblurring with Transformer

TL;DR

VDTR introduces a Transformer-based approach for video deblurring, leveraging long-range relation modeling to improve spatial and temporal restoration, outperforming CNN-based methods on multiple benchmarks.

Contribution

First adaptation of Transformer architecture for video deblurring, addressing non-uniform blurs, misalignment, and high computational costs with a hierarchical, window-based attention model.

Findings

Achieves competitive results on DVD, GOPRO, REDS, and BSD benchmarks.

Outperforms CNN-based methods in video deblurring tasks.

Demonstrates the effectiveness of Transformer for spatial and temporal modeling in videos.

Abstract

Video deblurring is still an unsolved problem due to the challenging spatio-temporal modeling process. While existing convolutional neural network-based methods show a limited capacity for effective spatial and temporal modeling for video deblurring. This paper presents VDTR, an effective Transformer-based model that makes the first attempt to adapt Transformer for video deblurring. VDTR exploits the superior long-range and relation modeling capabilities of Transformer for both spatial and temporal modeling. However, it is challenging to design an appropriate Transformer-based model for video deblurring due to the complicated non-uniform blurs, misalignment across multiple frames and the high computational costs for high-resolution spatial modeling. To address these problems, VDTR advocates performing attention within non-overlapping windows and exploiting the hierarchical structure for long-range dependencies modeling. For frame-level spatial modeling, we propose an encoder-decoder Transformer that utilizes multi-scale features for deblurring. For multi-frame temporal modeling, we adapt Transformer to fuse multiple spatial features efficiently. Compared with CNN-based methods, the proposed method achieves highly competitive results on both synthetic and real-world video deblurring benchmarks, including DVD, GOPRO, REDS and BSD. We hope such a Transformer-based architecture can serve as a powerful alternative baseline for video deblurring and other video restoration tasks. The source code will be available at \url{https://github.com/ljzycmd/VDTR}.