StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces

TL;DR

StochSync is a novel zero-shot method that combines diffusion synchronization and score distillation to generate high-quality images in arbitrary spaces with minimal conditioning, advancing 360-degree panoramas and 3D mesh texturing.

Contribution

It introduces StochSync, a new approach that unifies two existing methods, enhancing zero-shot image generation quality in arbitrary spaces with weak conditioning.

Findings

Outperforms previous finetuning methods in 360° panorama generation.

Achieves comparable results to existing methods in 3D mesh texturing.

Effectively generates images with minimal conditioning.

Abstract

We propose a zero-shot method for generating images in arbitrary spaces (e.g., a sphere for 360{\deg} panoramas and a mesh surface for texture) using a pretrained image diffusion model. The zero-shot generation of various visual content using a pretrained image diffusion model has been explored mainly in two directions. First, Diffusion Synchronization-performing reverse diffusion processes jointly across different projected spaces while synchronizing them in the target space-generates high-quality outputs when enough conditioning is provided, but it struggles in its absence. Second, Score Distillation Sampling-gradually updating the target space data through gradient descent-results in better coherence but often lacks detail. In this paper, we reveal for the first time the interconnection between these two methods while highlighting their differences. To this end, we propose StochSync, a novel approach that combines the strengths of both, enabling effective performance with weak conditioning. Our experiments demonstrate that StochSync provides the best performance in 360{\deg} panorama generation (where image conditioning is not given), outperforming previous finetuning-based methods, and also delivers comparable results in 3D mesh texturing (where depth conditioning is provided) with previous methods.