$\infty$-Brush: Controllable Large Image Synthesis with Diffusion Models in Infinite Dimensions

TL;DR

The paper introduces $ abla$-Brush, a novel diffusion model operating in infinite-dimensional function space, enabling controllable, high-resolution image synthesis at arbitrary sizes with preserved global structure and fine details.

Contribution

It presents the first conditional diffusion model in function space, overcoming resolution and scalability limitations of prior methods for large image synthesis.

Findings

Capable of synthesizing images up to 4096x4096 pixels.

Maintains global image structure while preserving fine details.

Outperforms traditional finite-dimensional and patch-based models.

Abstract

Synthesizing high-resolution images from intricate, domain-specific information remains a significant challenge in generative modeling, particularly for applications in large-image domains such as digital histopathology and remote sensing. Existing methods face critical limitations: conditional diffusion models in pixel or latent space cannot exceed the resolution on which they were trained without losing fidelity, and computational demands increase significantly for larger image sizes. Patch-based methods offer computational efficiency but fail to capture long-range spatial relationships due to their overreliance on local information. In this paper, we introduce a novel conditional diffusion model in infinite dimensions, $\infty$-Brush for controllable large image synthesis. We propose a cross-attention neural operator to enable conditioning in function space. Our model overcomes the constraints of traditional finite-dimensional diffusion models and patch-based methods, offering scalability and superior capability in preserving global image structures while maintaining fine details. To our best knowledge, $\infty$-Brush is the first conditional diffusion model in function space, that can controllably synthesize images at arbitrary resolutions of up to $4096\times4096$ pixels. The code is available at https://github.com/cvlab-stonybrook/infinity-brush.