High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion

TL;DR

The paper introduces HiFI, a patch-based cascaded diffusion model for high-resolution frame interpolation that effectively handles complex textures and motions while reducing memory usage and training costs.

Contribution

It proposes a novel single-resolution diffusion approach with patch processing for high-res interpolation, outperforming existing methods on challenging benchmarks.

Findings

HiFI achieves state-of-the-art results on multiple benchmarks.

The model effectively handles complex textures and large motions.

It reduces memory and training costs compared to multi-resolution cascades.

Abstract

Despite the recent progress, existing frame interpolation methods still struggle with processing extremely high resolution input and handling challenging cases such as repetitive textures, thin objects, and large motion. To address these issues, we introduce a patch-based cascaded pixel diffusion model for high resolution frame interpolation, HiFI, that excels in these scenarios while achieving competitive performance on standard benchmarks. Cascades, which generate a series of images from low to high resolution, can help significantly with large or complex motion that require both global context for a coarse solution and detailed context for high resolution output. However, contrary to prior work on cascaded diffusion models which perform diffusion on increasingly large resolutions, we use a single model that always performs diffusion at the same resolution and upsamples by processing patches of the inputs and the prior solution. At inference time, this drastically reduces memory usage and allows a single model, solving both frame interpolation (base model's task) and spatial up-sampling, saving training cost as well. HiFI excels at high-resolution images and complex repeated textures that require global context, achieving comparable or state-of-the-art performance on various benchmarks (Vimeo, Xiph, X-Test, and SEPE-8K). We further introduce a new dataset, LaMoR, that focuses on particularly challenging cases, and HiFI significantly outperforms other baselines. Please visit our project page for video results: https://hifi-diffusion.github.io