On the Limits of Latent Reuse in Diffusion Models

TL;DR

This paper investigates the reliability of reusing latent spaces in diffusion models under distribution shifts, providing theoretical insights into when such reuse is effective or requires shared representations.

Contribution

It offers a theoretical analysis of latent reuse limits in diffusion models, highlighting the roles of subspace misalignment and ambient noise.

Findings

Latent reuse reliability depends on principal-angle misalignment and noise amplification.

Reusing source latent spaces can induce score errors under distribution shift.

Shared latent dimensions are necessary depending on the geometric relationship of distributions.

Abstract

Diffusion models are often trained in low-dimensional latent spaces, which are then reused for related but shifted datasets. In this work, we study when such latent reuse remains reliable under distribution shift. We consider a source-target setting in which both datasets are approximately low-dimensional but may lie near different subspaces. We show that freezing and reusing a source latent space induces a target-domain score error governed by two quantities: the principal-angle misalignment between the source and target subspaces, and the target ambient noise amplified by the diffusion time scale. Motivated by these limits, we further study mixed source-target training and characterize how the required shared latent dimension depends on the relative geometry of the two distributions. Our results provide theoretical guidance on when latent reuse is reliable and when learning a shared representation may be necessary.