Progressive Training of A Two-Stage Framework for Video Restoration

TL;DR

This paper introduces a progressive two-stage training framework combining recurrent networks and transformers for video restoration, significantly improving training efficiency and performance in video super-resolution and quality enhancement tasks.

Contribution

It proposes a novel two-stage framework with specific training strategies like transfer learning and progressive training to enhance video restoration models.

Findings

Won two champion titles in NTIRE 2022 challenges

Reduced training time with proposed strategies

Achieved state-of-the-art performance in video restoration

Abstract

As a widely studied task, video restoration aims to enhance the quality of the videos with multiple potential degradations, such as noises, blurs and compression artifacts. Among video restorations, compressed video quality enhancement and video super-resolution are two of the main tacks with significant values in practical scenarios. Recently, recurrent neural networks and transformers attract increasing research interests in this field, due to their impressive capability in sequence-to-sequence modeling. However, the training of these models is not only costly but also relatively hard to converge, with gradient exploding and vanishing problems. To cope with these problems, we proposed a two-stage framework including a multi-frame recurrent network and a single-frame transformer. Besides, multiple training strategies, such as transfer learning and progressive training, are developed to shorten the training time and improve the model performance. Benefiting from the above technical contributions, our solution wins two champions and a runner-up in the NTIRE 2022 super-resolution and quality enhancement of compressed video challenges. Code is available at https://github.com/ryanxingql/winner-ntire22-vqe.