VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models

TL;DR

This paper introduces VISION-XL, a high-definition video inverse problem solver using latent image diffusion models, achieving fast, high-quality reconstructions with improved temporal consistency on a single GPU.

Contribution

The paper presents a novel latent-space diffusion framework with efficient sampling and initialization techniques for high-res video inverse problems, outperforming prior methods.

Findings

Achieves HD video reconstruction over 1280x720 in under 6 seconds per frame.

Supports multiple aspect ratios and complex spatial degradations.

Provides state-of-the-art results across various inverse problems.

Abstract

In this paper, we propose a novel framework for solving high-definition video inverse problems using latent image diffusion models. Building on recent advancements in spatio-temporal optimization for video inverse problems using image diffusion models, our approach leverages latent-space diffusion models to achieve enhanced video quality and resolution. To address the high computational demands of processing high-resolution frames, we introduce a pseudo-batch consistent sampling strategy, allowing efficient operation on a single GPU. Additionally, to improve temporal consistency, we present pseudo-batch inversion, an initialization technique that incorporates informative latents from the measurement. By integrating with SDXL, our framework achieves state-of-the-art video reconstruction across a wide range of spatio-temporal inverse problems, including complex combinations of frame averaging and various spatial degradations, such as deblurring, super-resolution, and inpainting. Unlike previous methods, our approach supports multiple aspect ratios (landscape, vertical, and square) and delivers HD-resolution reconstructions (exceeding 1280x720) in under 6 seconds per frame on a single NVIDIA 4090 GPU.