Solving Video Inverse Problems Using Image Diffusion Models

TL;DR

This paper introduces a novel video inverse problem solver that leverages image diffusion models by treating video frames as batch data, enabling effective spatio-temporal restoration without training dedicated video diffusion models.

Contribution

The method innovatively applies image diffusion models to videos by treating the temporal dimension as batch data and introduces a batch-consistent diffusion sampling strategy for improved spatio-temporal consistency.

Findings

Achieves state-of-the-art results on various video inverse problems.

Effectively restores videos with complex spatio-temporal degradations.

Demonstrates efficiency by avoiding training dedicated video diffusion models.

Abstract

Recently, diffusion model-based inverse problem solvers (DIS) have emerged as state-of-the-art approaches for addressing inverse problems, including image super-resolution, deblurring, inpainting, etc. However, their application to video inverse problems arising from spatio-temporal degradation remains largely unexplored due to the challenges in training video diffusion models. To address this issue, here we introduce an innovative video inverse solver that leverages only image diffusion models. Specifically, by drawing inspiration from the success of the recent decomposed diffusion sampler (DDS), our method treats the time dimension of a video as the batch dimension of image diffusion models and solves spatio-temporal optimization problems within denoised spatio-temporal batches derived from each image diffusion model. Moreover, we introduce a batch-consistent diffusion sampling strategy that encourages consistency across batches by synchronizing the stochastic noise components in image diffusion models. Our approach synergistically combines batch-consistent sampling with simultaneous optimization of denoised spatio-temporal batches at each reverse diffusion step, resulting in a novel and efficient diffusion sampling strategy for video inverse problems. Experimental results demonstrate that our method effectively addresses various spatio-temporal degradations in video inverse problems, achieving state-of-the-art reconstructions. Project page: https://svi-diffusion.github.io/