VRT: A Video Restoration Transformer

TL;DR

This paper introduces VRT, a novel Video Restoration Transformer that effectively models long-range temporal dependencies and aligns frames using a parallel warping approach, significantly improving performance across multiple video restoration tasks.

Contribution

VRT is the first transformer-based model to incorporate parallel frame prediction and long-range temporal modeling for comprehensive video restoration.

Findings

Outperforms state-of-the-art methods by up to 2.16dB in PSNR.

Effective across five video restoration tasks on fourteen datasets.

Demonstrates superior long-range temporal dependency modeling.

Abstract

Video restoration (e.g., video super-resolution) aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which either is restricted by frame-by-frame restoration or lacks long-range modelling ability. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction and long-range temporal dependency modelling abilities. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal mutual self attention (TMSA) and parallel warping. TMSA divides the video into small clips, on which mutual attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on five tasks, including video super-resolution, video deblurring, video denoising, video frame interpolation and space-time video super-resolution, demonstrate that VRT outperforms the state-of-the-art methods by large margins ($\textbf{up to 2.16dB}$) on fourteen benchmark datasets.