TanDiT: Tangent-Plane Diffusion Transformer for High-Quality 360{\deg} Panorama Generation

TL;DR

TanDiT introduces a unified diffusion approach for high-quality 360-degree panoramic image synthesis, effectively addressing geometric distortion and loop consistency challenges with post-processing and new evaluation metrics.

Contribution

The paper presents TanDiT, a novel tangent-plane diffusion transformer that generates panoramic images in a single iteration and introduces metrics for panoramic quality assessment.

Findings

Effective generalization beyond training data

Robust interpretation of complex text prompts

Seamless integration with various generative models

Abstract

Recent advances in image generation have led to remarkable improvements in synthesizing perspective images. However, these models still struggle with panoramic image generation due to unique challenges, including varying levels of geometric distortion and the requirement for seamless loop-consistency. To address these issues while leveraging the strengths of the existing models, we introduce TanDiT, a method that synthesizes panoramic scenes by generating grids of tangent-plane images covering the entire 360$^\circ$ view. Unlike previous methods relying on multiple diffusion branches, TanDiT utilizes a unified diffusion model trained to produce these tangent-plane images simultaneously within a single denoising iteration. Furthermore, we propose a model-agnostic post-processing step specifically designed to enhance global coherence across the generated panoramas. To accurately assess panoramic image quality, we also present two specialized metrics, TangentIS and TangentFID, and provide a comprehensive benchmark comprising captioned panoramic datasets and standardized evaluation scripts. Extensive experiments demonstrate that our method generalizes effectively beyond its training data, robustly interprets detailed and complex text prompts, and seamlessly integrates with various generative models to yield high-quality, diverse panoramic images.