DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

TL;DR

DiffIR2VR-Zero is a versatile zero-shot video restoration framework that leverages pre-trained image diffusion models, ensuring high-quality, temporally consistent results across various degradation scenarios without additional training.

Contribution

It introduces a hierarchical latent warping and hybrid token merging mechanism, enabling effective video restoration with existing image diffusion models without retraining.

Findings

Achieves superior temporal consistency across diverse datasets.

Handles challenging scenarios like super-resolution and severe noise.

Works with any pre-trained image diffusion model without modifications.

Abstract

We present DiffIR2VR-Zero, a zero-shot framework that enables any pre-trained image restoration diffusion model to perform high-quality video restoration without additional training. While image diffusion models have shown remarkable restoration capabilities, their direct application to video leads to temporal inconsistencies, and existing video restoration methods require extensive retraining for different degradation types. Our approach addresses these challenges through two key innovations: a hierarchical latent warping strategy that maintains consistency across both keyframes and local frames, and a hybrid token merging mechanism that adaptively combines optical flow and feature matching. Through extensive experiments, we demonstrate that our method not only maintains the high-quality restoration of base diffusion models but also achieves superior temporal consistency across diverse datasets and degradation conditions, including challenging scenarios like 8$\times$ super-resolution and severe noise. Importantly, our framework works with any image restoration diffusion model, providing a versatile solution for video enhancement without task-specific training or modifications. Project page: https://jimmycv07.github.io/DiffIR2VR_web/