JoPano: Unified Panorama Generation via Joint Modeling

TL;DR

JoPano introduces a unified model for panorama generation that combines text-to-panorama and view-to-panorama tasks using a DiT-based architecture, improving quality and efficiency with novel adapters and blending techniques.

Contribution

The paper presents a joint modeling framework with a new adapter and blending methods, enabling high-quality panorama generation for multiple tasks within a single model.

Findings

Achieves state-of-the-art results on FID, CLIP-FID, IS, and CLIP-Score metrics.

Effectively reduces seam inconsistencies with Poisson Blending.

Unifies two panorama generation tasks in a single, efficient model.

Abstract

Panorama generation has recently attracted growing interest in the research community, with two core tasks, text-to-panorama and view-to-panorama generation. However, existing methods still face two major challenges: their U-Net-based architectures constrain the visual quality of the generated panoramas, and they usually treat the two core tasks independently, which leads to modeling redundancy and inefficiency. To overcome these challenges, we propose a joint-face panorama (JoPano) generation approach that unifies the two core tasks within a DiT-based model. To transfer the rich generative capabilities of existing DiT backbones learned from natural images to the panorama domain, we propose a Joint-Face Adapter built on the cubemap representation of panoramas, which enables a pretrained DiT to jointly model and generate different views of a panorama. We further apply Poisson Blending to reduce seam inconsistencies that often appear at the boundaries between cube faces. Correspondingly, we introduce Seam-SSIM and Seam-Sobel metrics to quantitatively evaluate the seam consistency. Moreover, we propose a condition switching mechanism that unifies text-to-panorama and view-to-panorama tasks within a single model. Comprehensive experiments show that JoPano can generate high-quality panoramas for both text-to-panorama and view-to-panorama generation tasks, achieving state-of-the-art performance on FID, CLIP-FID, IS, and CLIP-Score metrics.