Mitigating Artifacts in Real-World Video Super-Resolution Models

TL;DR

This paper introduces FastRealVSR, a novel video super-resolution method that uses a Hidden State Attention module to reduce artifacts and improve speed in real-world scenarios with complex degradations.

Contribution

The paper proposes a Hidden State Attention module with a Selective Cross Attention mechanism to mitigate artifacts and enhance efficiency in real-world video super-resolution models.

Findings

Achieves 2x speedup over previous methods

Produces higher quality super-resolved videos

Effectively reduces artifacts in complex degradations

Abstract

The recurrent structure is a prevalent framework for the task of video super-resolution, which models the temporal dependency between frames via hidden states. When applied to real-world scenarios with unknown and complex degradations, hidden states tend to contain unpleasant artifacts and propagate them to restored frames. In this circumstance, our analyses show that such artifacts can be largely alleviated when the hidden state is replaced with a cleaner counterpart. Based on the observations, we propose a Hidden State Attention (HSA) module to mitigate artifacts in real-world video super-resolution. Specifically, we first adopt various cheap filters to produce a hidden state pool. For example, Gaussian blur filters are for smoothing artifacts while sharpening filters are for enhancing details. To aggregate a new hidden state that contains fewer artifacts from the hidden state pool, we devise a Selective Cross Attention (SCA) module, in which the attention between input features and each hidden state is calculated. Equipped with HSA, our proposed method, namely FastRealVSR, is able to achieve 2x speedup while obtaining better performance than Real-BasicVSR. Codes will be available at https://github.com/TencentARC/FastRealVSR