TDM: Temporally-Consistent Diffusion Model for All-in-One Real-World Video Restoration

TL;DR

This paper introduces TDM, a unified diffusion-based model that effectively restores various degraded videos by leveraging pre-trained diffusion models and innovative inference strategies, achieving superior quality and consistency.

Contribution

The paper presents a novel all-in-one video restoration framework using diffusion models, with a new training and inference strategy for diverse real-world video restoration tasks.

Findings

Outperforms state-of-the-art methods in multiple restoration tasks.

Demonstrates strong generalization to real-world videos.

Maintains high temporal consistency across frames.

Abstract

In this paper, we propose the first diffusion-based all-in-one video restoration method that utilizes the power of a pre-trained Stable Diffusion and a fine-tuned ControlNet. Our method can restore various types of video degradation with a single unified model, overcoming the limitation of standard methods that require specific models for each restoration task. Our contributions include an efficient training strategy with Task Prompt Guidance (TPG) for diverse restoration tasks, an inference strategy that combines Denoising Diffusion Implicit Models~(DDIM) inversion with a novel Sliding Window Cross-Frame Attention (SW-CFA) mechanism for enhanced content preservation and temporal consistency, and a scalable pipeline that makes our method all-in-one to adapt to different video restoration tasks. Through extensive experiments on five video restoration tasks, we demonstrate the superiority of our method in generalization capability to real-world videos and temporal consistency preservation over existing state-of-the-art methods. Our method advances the video restoration task by providing a unified solution that enhances video quality across multiple applications.