Coevolving Representations in Joint Image-Feature Diffusion

TL;DR

This paper introduces CoReDi, a framework where semantic representations evolve during training to enhance diffusion-based image generation, leading to faster convergence and better quality.

Contribution

It proposes a novel coevolving representation method that adapts semantic spaces during training, improving diffusion models over fixed representations.

Findings

CoReDi achieves faster convergence in diffusion training.

It results in higher quality generated images.

Adaptive semantic spaces better complement image latents.

Abstract

Joint image-feature generative modeling has recently emerged as an effective strategy for improving diffusion training by coupling low-level VAE latents with high-level semantic features extracted from pre-trained visual encoders. However, existing approaches rely on a fixed representation space, constructed independently of the generative objective and kept unchanged during training. We argue that the representation space guiding diffusion should itself adapt to the generative task. To this end, we propose Coevolving Representation Diffusion (CoReDi), a framework in which the semantic representation space evolves during training by learning a lightweight linear projection jointly with the diffusion model. While naively optimizing this projection leads to degenerate solutions, we show that stable coevolution can be achieved through a combination of stop-gradient targets, normalization, and targeted regularization that prevents feature collapse. This formulation enables the semantic space to progressively specialize to the needs of image synthesis, improving its complementarity with image latents. We apply CoReDi to both VAE latent diffusion and pixel-space diffusion, demonstrating that adaptive semantic representations improve generative modeling across both settings. Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.