RealisVSR: Detail-enhanced Diffusion for Real-World 4K Video Super-Resolution

TL;DR

RealisVSR introduces a novel diffusion-based approach with enhanced detail recovery and a new 4K benchmark, significantly improving real-world 4K video super-resolution performance.

Contribution

The paper presents a high-frequency detail-enhanced diffusion model with a new architecture, loss function, and 4K benchmark for improved real-world video super-resolution.

Findings

Outperforms existing methods on multiple benchmarks

Requires only 5-25% of training data of previous approaches

Excels in ultra-high-resolution video super-resolution

Abstract

Video Super-Resolution (VSR) has achieved significant progress through diffusion models, effectively addressing the over-smoothing issues inherent in GAN-based methods. Despite recent advances, three critical challenges persist in VSR community: 1) Inconsistent modeling of temporal dynamics in foundational models; 2) limited high-frequency detail recovery under complex real-world degradations; and 3) insufficient evaluation of detail enhancement and 4K super-resolution, as current methods primarily rely on 720P datasets with inadequate details. To address these challenges, we propose RealisVSR, a high-frequency detail-enhanced video diffusion model with three core innovations: 1) Consistency Preserved ControlNet (CPC) architecture integrated with the Wan2.1 video diffusion to model the smooth and complex motions and suppress artifacts; 2) High-Frequency Rectified Diffusion Loss (HR-Loss) combining wavelet decomposition and HOG feature constraints for texture restoration; 3) RealisVideo-4K, the first public 4K VSR benchmark containing 1,000 high-definition video-text pairs. Leveraging the advanced spatio-temporal guidance of Wan2.1, our method requires only 5-25% of the training data volume compared to existing approaches. Extensive experiments on VSR benchmarks (REDS, SPMCS, UDM10, YouTube-HQ, VideoLQ, RealisVideo-720P) demonstrate our superiority, particularly in ultra-high-resolution scenarios.