STCDiT: Spatio-Temporally Consistent Diffusion Transformer for High-Quality Video Super-Resolution

TL;DR

STCDiT is a novel video super-resolution framework that combines motion-aware reconstruction and anchor-frame guidance within a diffusion model to produce high-quality, structurally faithful, and temporally stable videos from degraded inputs.

Contribution

The paper introduces a new approach integrating motion-aware VAE reconstruction and anchor-frame guidance into a diffusion model for improved video super-resolution.

Findings

Outperforms state-of-the-art methods in structural fidelity.

Achieves superior temporal consistency in reconstructed videos.

Effectively handles complex camera motions.

Abstract

We present STCDiT, a video super-resolution framework built upon a pre-trained video diffusion model, aiming to restore structurally faithful and temporally stable videos from degraded inputs, even under complex camera motions. The main challenges lie in maintaining temporal stability during reconstruction and preserving structural fidelity during generation. To address these challenges, we first develop a motion-aware VAE reconstruction method that performs segment-wise reconstruction, with each segment clip exhibiting uniform motion characteristic, thereby effectively handling videos with complex camera motions. Moreover, we observe that the first-frame latent extracted by the VAE encoder in each clip, termed the anchor-frame latent, remains unaffected by temporal compression and retains richer spatial structural information than subsequent frame latents. We further develop an anchor-frame guidance approach that leverages structural information from anchor frames to constrain the generation process and improve structural fidelity of video features. Coupling these two designs enables the video diffusion model to achieve high-quality video super-resolution. Extensive experiments show that STCDiT outperforms state-of-the-art methods in terms of structural fidelity and temporal consistency.