Decoupling Complexity from Scale in Latent Diffusion Model

TL;DR

DCS-LDM introduces a hierarchical, scale-independent latent space for visual generation, decoupling content complexity from scale to enable flexible, high-quality image and video synthesis at various resolutions and frame rates.

Contribution

It proposes a novel hierarchical latent space that models content complexity independently of scale, allowing flexible resolution and frame rate generation in latent diffusion models.

Findings

Achieves performance comparable to state-of-the-art methods.

Supports arbitrary resolutions and frame rates within a fixed latent space.

Enables progressive coarse-to-fine generation.

Abstract

Existing latent diffusion models typically couple scale with content complexity, using more latent tokens to represent higher-resolution images or higher-frame rate videos. However, the latent capacity required to represent visual data primarily depends on content complexity, with scale serving only as an upper bound. Motivated by this observation, we propose DCS-LDM, a novel paradigm for visual generation that decouples information complexity from scale. DCS-LDM constructs a hierarchical, scale-independent latent space that models sample complexity through multi-level tokens and supports decoding to arbitrary resolutions and frame rates within a fixed latent representation. This latent space enables DCS-LDM to achieve a flexible computation-quality tradeoff. Furthermore, by decomposing structural and detailed information across levels, DCS-LDM supports a progressive coarse-to-fine generation paradigm. Experimental results show that DCS-LDM delivers performance comparable to state-of-the-art methods while offering flexible generation across diverse scales and visual qualities.