Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation

TL;DR

This paper introduces a novel neural network architecture trained as a denoising diffusion model that simultaneously generates and segments images without supervision, leveraging a computational bottleneck to partition and denoise regions in parallel.

Contribution

The proposed architecture enables unsupervised image generation and segmentation using a diffusion model with a built-in partitioning mechanism, eliminating the need for annotated data.

Findings

Achieves accurate unsupervised segmentation of real images.

Generates high-quality synthetic images.

Works across multiple datasets without fine-tuning.

Abstract

We develop a neural network architecture which, trained in an unsupervised manner as a denoising diffusion model, simultaneously learns to both generate and segment images. Learning is driven entirely by the denoising diffusion objective, without any annotation or prior knowledge about regions during training. A computational bottleneck, built into the neural architecture, encourages the denoising network to partition an input into regions, denoise them in parallel, and combine the results. Our trained model generates both synthetic images and, by simple examination of its internal predicted partitions, a semantic segmentation of those images. Without any finetuning, we directly apply our unsupervised model to the downstream task of segmenting real images via noising and subsequently denoising them. Experiments demonstrate that our model achieves accurate unsupervised image segmentation and high-quality synthetic image generation across multiple datasets.