ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schr\"odinger Bridge

TL;DR

ToddlerDiffusion introduces a modality-space diffusion framework that cascades interpretable stages for high-quality image generation, leveraging Schr"odinger Bridge for optimal transport, resulting in improved efficiency and interpretability over existing methods.

Contribution

The paper presents a novel cascaded diffusion approach in modality space using Schr"odinger Bridge, enhancing interpretability, efficiency, and performance in image generation.

Findings

Outperforms state-of-the-art methods on multiple datasets.

Operates twice as fast as LDM with smaller architecture.

Achieves high-quality image generation with interpretable stages.

Abstract

Diffusion models break down the challenging task of generating data from high-dimensional distributions into a series of easier denoising steps. Inspired by this paradigm, we propose a novel approach that extends the diffusion framework into modality space, decomposing the complex task of RGB image generation into simpler, interpretable stages. Our method, termed ToddlerDiffusion, cascades modality-specific models, each responsible for generating an intermediate representation, such as contours, palettes, and detailed textures, ultimately culminating in a high-quality RGB image. Instead of relying on the naive LDM concatenation conditioning mechanism to connect the different stages together, we employ Schr\"odinger Bridge to determine the optimal transport between different modalities. Although employing a cascaded pipeline introduces more stages, which could lead to a more complex architecture, each stage is meticulously formulated for efficiency and accuracy, surpassing Stable-Diffusion (LDM) performance. Modality composition not only enhances overall performance but enables emerging proprieties such as consistent editing, interaction capabilities, high-level interpretability, and faster convergence and sampling rate. Extensive experiments on diverse datasets, including LSUN-Churches, ImageNet, CelebHQ, and LAION-Art, demonstrate the efficacy of our approach, consistently outperforming state-of-the-art methods. For instance, ToddlerDiffusion achieves notable efficiency, matching LDM performance on LSUN-Churches while operating 2$\times$ faster with a 3$\times$ smaller architecture. The project website is available at: https://toddlerdiffusion.github.io/website/