Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas

TL;DR

This paper introduces the Merge-Attend-Diffuse operator to enhance semantic coherence in panorama images generated by diffusion models, addressing incoherence and diversity trade-offs in joint diffusion methods.

Contribution

We propose a novel operator that improves semantic coherence in panorama diffusion by merging diffusion paths and reprogramming attention mechanisms within pretrained models.

Findings

Enhanced semantic coherence in panorama images.

Maintained visual quality and prompt compatibility.

Validated through extensive experiments and user study.

Abstract

Diffusion models have become the State-of-the-Art for text-to-image generation, and increasing research effort has been dedicated to adapting the inference process of pretrained diffusion models to achieve zero-shot capabilities. An example is the generation of panorama images, which has been tackled in recent works by combining independent diffusion paths over overlapping latent features, which is referred to as joint diffusion, obtaining perceptually aligned panoramas. However, these methods often yield semantically incoherent outputs and trade-off diversity for uniformity. To overcome this limitation, we propose the Merge-Attend-Diffuse operator, which can be plugged into different types of pretrained diffusion models used in a joint diffusion setting to improve the perceptual and semantical coherence of the generated panorama images. Specifically, we merge the diffusion paths, reprogramming self- and cross-attention to operate on the aggregated latent space. Extensive quantitative and qualitative experimental analysis, together with a user study, demonstrate that our method maintains compatibility with the input prompt and visual quality of the generated images while increasing their semantic coherence. We release the code at https://github.com/aimagelab/MAD.