FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

TL;DR

FrescoDiffusion is a training-free method that enhances 4K image-to-video generation by combining tiled denoising with a global latent prior, ensuring high-resolution detail and spatial-temporal coherence.

Contribution

It introduces a novel, training-free approach for large-format I2V generation that fuses tiled denoising with a precomputed global latent reference for improved coherence.

Findings

Improved global consistency and fidelity over tiled baselines.

Efficient 4K image-to-video generation with fine detail preservation.

Enables controllable trade-off between creativity and consistency.

Abstract

Diffusion-based image-to-video (I2V) models are increasingly effective, yet they struggle to scale to ultra-high-resolution inputs (e.g., 4K). Generating videos at the model's native resolution often loses fine-grained structure, whereas high-resolution tiled denoising preserves local detail but breaks global layout consistency. This failure mode is particularly severe in the fresco animation setting: monumental artworks containing many distinct characters, objects, and semantically different sub-scenes that must remain spatially coherent over time. We introduce FrescoDiffusion, a training-free method for coherent large-format I2V generation from a single complex image. The key idea is to augment tiled denoising with a precomputed latent prior: we first generate a low-resolution video at the underlying model resolution and upsample its latent trajectory to obtain a global reference that captures long-range temporal and spatial structure. For 4K generation, we compute per-tile noise predictions and fuse them with this reference at every diffusion timestep by minimizing a single weighted least-squares objective in model-output space. The objective combines a standard tile-merging criterion with our regularization term, yielding a closed-form fusion update that strengthens global coherence while retaining fine detail. We additionally provide a spatial regularization variable that enables region-level control over where motion is allowed. Experiments on the VBench-I2V dataset and our proposed fresco I2V dataset show improved global consistency and fidelity over tiled baselines, while being computationally efficient. Our regularization enables explicit controllability of the trade-off between creativity and consistency.