What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?

TL;DR

This paper analyzes how stable diffusion models adapt to 360-degree panorama generation, revealing key mechanisms and introducing UniPano, a simple, efficient baseline that outperforms existing methods.

Contribution

It provides insights into the internal mechanisms of diffusion models for panoramas and proposes UniPano, a novel, scalable framework that improves performance and efficiency.

Findings

Attention query and key matrices share common information across domains.

Value and output matrices specialize in panoramic adaptation.

UniPano outperforms existing methods in quality and efficiency.

Abstract

Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion, has stimulated research to adapt them to 360-degree panorama generation. Prior work has demonstrated the feasibility of using conventional low-rank adaptation techniques on pre-trained diffusion models to generate panoramic images. However, the substantial domain gap between perspective and panoramic images raises questions about the underlying mechanisms enabling this empirical success. We hypothesize and examine that the trainable counterparts exhibit distinct behaviors when fine-tuned on panoramic data, and such an adaptation conceals some intrinsic mechanism to leverage the prior knowledge within the pre-trained diffusion models. Our analysis reveals the following: 1) the query and key matrices in the attention modules are responsible for common information that can be shared between the panoramic and perspective domains, thus are less relevant to panorama generation; and 2) the value and output weight matrices specialize in adapting pre-trained knowledge to the panoramic domain, playing a more critical role during fine-tuning for panorama generation. We empirically verify these insights by introducing a simple framework called UniPano, with the objective of establishing an elegant baseline for future research. UniPano not only outperforms existing methods but also significantly reduces memory usage and training time compared to prior dual-branch approaches, making it scalable for end-to-end panorama generation with higher resolution. The code will be released.