DAM-VSR: Disentanglement of Appearance and Motion for Video Super-Resolution

TL;DR

DAM-VSR introduces a novel framework that disentangles appearance and motion to improve the quality and temporal consistency of video super-resolution, leveraging diffusion models and reference-based enhancement.

Contribution

It proposes a new appearance and motion disentanglement framework for VSR, combining diffusion models with reference image super-resolution and motion control for better detail and consistency.

Findings

Achieves state-of-the-art performance on real-world and AIGC data.

Effectively generates detailed and temporally consistent video frames.

Utilizes a motion-aligned bidirectional sampling strategy for longer videos.

Abstract

Real-world video super-resolution (VSR) presents significant challenges due to complex and unpredictable degradations. Although some recent methods utilize image diffusion models for VSR and have shown improved detail generation capabilities, they still struggle to produce temporally consistent frames. We attempt to use Stable Video Diffusion (SVD) combined with ControlNet to address this issue. However, due to the intrinsic image-animation characteristics of SVD, it is challenging to generate fine details using only low-quality videos. To tackle this problem, we propose DAM-VSR, an appearance and motion disentanglement framework for VSR. This framework disentangles VSR into appearance enhancement and motion control problems. Specifically, appearance enhancement is achieved through reference image super-resolution, while motion control is achieved through video ControlNet. This disentanglement fully leverages the generative prior of video diffusion models and the detail generation capabilities of image super-resolution models. Furthermore, equipped with the proposed motion-aligned bidirectional sampling strategy, DAM-VSR can conduct VSR on longer input videos. DAM-VSR achieves state-of-the-art performance on real-world data and AIGC data, demonstrating its powerful detail generation capabilities.