Nested Diffusion Models Using Hierarchical Latent Priors

TL;DR

This paper presents nested diffusion models that use hierarchical latent priors to improve image generation quality, especially for complex scenes, by progressively generating and conditioning on semantic-level latent variables.

Contribution

The paper introduces a hierarchical diffusion framework with semantic latent variables, leveraging a pre-trained encoder, to enhance image quality with minimal additional computational cost.

Findings

Significant improvement in image quality across multiple datasets.

Outperforms baseline conditional systems in unconditional generation.

Efficient hierarchical approach with low overhead.

Abstract

We introduce nested diffusion models, an efficient and powerful hierarchical generative framework that substantially enhances the generation quality of diffusion models, particularly for images of complex scenes. Our approach employs a series of diffusion models to progressively generate latent variables at different semantic levels. Each model in this series is conditioned on the output of the preceding higher-level models, culminating in image generation. Hierarchical latent variables guide the generation process along predefined semantic pathways, allowing our approach to capture intricate structural details while significantly improving image quality. To construct these latent variables, we leverage a pre-trained visual encoder, which learns strong semantic visual representations, and modulate its capacity via dimensionality reduction and noise injection. Across multiple datasets, our system demonstrates significant enhancements in image quality for both unconditional and class/text conditional generation. Moreover, our unconditional generation system substantially outperforms the baseline conditional system. These advancements incur minimal computational overhead as the more abstract levels of our hierarchy work with lower-dimensional representations.