Investigating Tradeoffs in Real-World Video Super-Resolution

TL;DR

This paper introduces RealBasicVSR, a real-world video super-resolution method that employs image pre-cleaning and a stochastic degradation scheme to improve quality and training efficiency, along with a new dataset for benchmarking.

Contribution

The paper presents a novel cleaning module for VSR, a stochastic degradation training scheme, and the VideoLQ dataset for fair benchmarking of real-world VSR methods.

Findings

Pre-cleaning reduces artifacts and improves VSR quality.

Stochastic degradation scheme cuts training time by 40%.

Longer sequences during training enhance temporal information utilization.

Abstract

The diversity and complexity of degradations in real-world video super-resolution (VSR) pose non-trivial challenges in inference and training. First, while long-term propagation leads to improved performance in cases of mild degradations, severe in-the-wild degradations could be exaggerated through propagation, impairing output quality. To balance the tradeoff between detail synthesis and artifact suppression, we found an image pre-cleaning stage indispensable to reduce noises and artifacts prior to propagation. Equipped with a carefully designed cleaning module, our RealBasicVSR outperforms existing methods in both quality and efficiency. Second, real-world VSR models are often trained with diverse degradations to improve generalizability, requiring increased batch size to produce a stable gradient. Inevitably, the increased computational burden results in various problems, including 1) speed-performance tradeoff and 2) batch-length tradeoff. To alleviate the first tradeoff, we propose a stochastic degradation scheme that reduces up to 40\% of training time without sacrificing performance. We then analyze different training settings and suggest that employing longer sequences rather than larger batches during training allows more effective uses of temporal information, leading to more stable performance during inference. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences containing rich textures and patterns. Our dataset can serve as a common ground for benchmarking. Code, models, and the dataset will be made publicly available.