Learned representation-guided diffusion models for large-image generation

TL;DR

This paper introduces a novel diffusion model conditioned on self-supervised learning embeddings to generate high-quality, large-scale images in specialized domains like histopathology and satellite imagery, without requiring extensive manual annotations.

Contribution

The authors propose a new SSL-guided diffusion approach that enables high-fidelity large-image synthesis and robust generalization across datasets, even with embeddings from different sources or modalities.

Findings

Effective generation of high-quality histopathology and satellite images.

Improved classifier accuracy through data augmentation with generated images.

Successful text-to-large image synthesis demonstrating versatility.

Abstract

To synthesize high-fidelity samples, diffusion models typically require auxiliary data to guide the generation process. However, it is impractical to procure the painstaking patch-level annotation effort required in specialized domains like histopathology and satellite imagery; it is often performed by domain experts and involves hundreds of millions of patches. Modern-day self-supervised learning (SSL) representations encode rich semantic and visual information. In this paper, we posit that such representations are expressive enough to act as proxies to fine-grained human labels. We introduce a novel approach that trains diffusion models conditioned on embeddings from SSL. Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. In addition, we construct larger images by assembling spatially consistent patches inferred from SSL embeddings, preserving long-range dependencies. Augmenting real data by generating variations of real images improves downstream classifier accuracy for patch-level and larger, image-scale classification tasks. Our models are effective even on datasets not encountered during training, demonstrating their robustness and generalizability. Generating images from learned embeddings is agnostic to the source of the embeddings. The SSL embeddings used to generate a large image can either be extracted from a reference image, or sampled from an auxiliary model conditioned on any related modality (e.g. class labels, text, genomic data). As proof of concept, we introduce the text-to-large image synthesis paradigm where we successfully synthesize large pathology and satellite images out of text descriptions.