Spherical Dense Text-to-Image Synthesis

TL;DR

This paper introduces a unified approach for spherical dense text-to-image synthesis by integrating existing dense T2I models with panorama models, and presents new datasets and methods to improve image quality and layout adherence.

Contribution

It proposes MultiStitchDiffusion and MultiPanFusion methods for spherical T2I, and creates the DSynView dataset for evaluation, advancing the state of spherical image synthesis.

Findings

MSTD outperforms MPF in image quality and prompt adherence

MPF generates more diverse images but has foreground synthesis issues

Proposed improvements include bootstrap-coupling and perspective attention adjustments.

Abstract

Recent advancements in text-to-image (T2I) have improved synthesis results, but challenges remain in layout control and generating omnidirectional panoramic images. Dense T2I (DT2I) and spherical T2I (ST2I) models address these issues, but so far no unified approach exists. Trivial approaches, like prompting a DT2I model to generate panoramas can not generate proper spherical distortions and seamless transitions at the borders. Our work shows that spherical dense text-to-image (SDT2I) can be achieved by integrating training-free DT2I approaches into finetuned panorama models. Specifically, we propose MultiStitchDiffusion (MSTD) and MultiPanFusion (MPF) by integrating MultiDiffusion into StitchDiffusion and PanFusion, respectively. Since no benchmark for SDT2I exists, we further construct Dense-Synthetic-View (DSynView), a new synthetic dataset containing spherical layouts to evaluate our models. Our results show that MSTD outperforms MPF across image quality as well as prompt- and layout adherence. MultiPanFusion generates more diverse images but struggles to synthesize flawless foreground objects. We propose bootstrap-coupling and turning off equirectangular perspective-projection attention in the foreground as an improvement of MPF. Link to code https://github.com/sdt2i/spherical-dense-text-to-image