PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance

TL;DR

PanoFree is a tuning-free method for multi-view panorama image generation that enhances consistency and quality by iterative warping and inpainting with cross-view guidance, avoiding extensive fine-tuning.

Contribution

It introduces a novel tuning-free approach supporting complex correspondences, with iterative warping, inpainting, and cross-view guidance to improve multi-view panorama generation.

Findings

Significant error reduction in multi-view panorama generation

Up to 5x faster and 3x more memory-efficient than existing methods

Superior diversity and global consistency in generated images

Abstract

Immersive scene generation, notably panorama creation, benefits significantly from the adaptation of large pre-trained text-to-image (T2I) models for multi-view image generation. Due to the high cost of acquiring multi-view images, tuning-free generation is preferred. However, existing methods are either limited to simple correspondences or require extensive fine-tuning to capture complex ones. We present PanoFree, a novel method for tuning-free multi-view image generation that supports an extensive array of correspondences. PanoFree sequentially generates multi-view images using iterative warping and inpainting, addressing the key issues of inconsistency and artifacts from error accumulation without the need for fine-tuning. It improves error accumulation by enhancing cross-view awareness and refines the warping and inpainting processes via cross-view guidance, risky area estimation and erasing, and symmetric bidirectional guided generation for loop closure, alongside guidance-based semantic and density control for scene structure preservation. In experiments on Planar, 360{\deg}, and Full Spherical Panoramas, PanoFree demonstrates significant error reduction, improves global consistency, and boosts image quality without extra fine-tuning. Compared to existing methods, PanoFree is up to 5x more efficient in time and 3x more efficient in GPU memory usage, and maintains superior diversity of results (2x better in our user study). PanoFree offers a viable alternative to costly fine-tuning or the use of additional pre-trained models. Project website at https://panofree.github.io/.