Canonical Latent Representations in Conditional Diffusion Models
Yitao Xu, Tong Zhang, Ehsan Pajouheshgar, Sabine S\"usstrunk
TL;DR
This paper introduces Canonical Latent Representations (CLAReps) in conditional diffusion models to extract robust, interpretable class features, enabling effective knowledge transfer and improved adversarial robustness in downstream classifiers.
Contribution
The paper proposes CLAReps to disentangle class features from irrelevant signals in CDMs and introduces CaDistill, a diffusion-based knowledge transfer method using minimal data.
Findings
CLAReps produce representative, interpretable class samples.
CaDistill transfers core class knowledge using only 10% of training data.
Students trained with CLAReps show enhanced robustness and generalization.
Abstract
Conditional diffusion models (CDMs) have shown impressive performance across a range of generative tasks. Their ability to model the full data distribution has opened new avenues for analysis-by-synthesis in downstream discriminative learning. However, this same modeling capacity causes CDMs to entangle the class-defining features with irrelevant context, posing challenges to extracting robust and interpretable representations. To this end, we identify Canonical LAtent Representations (CLAReps), latent codes whose internal CDM features preserve essential categorical information while discarding non-discriminative signals. When decoded, CLAReps produce representative samples for each class, offering an interpretable and compact summary of the core class semantics with minimal irrelevant details. Exploiting CLAReps, we develop a novel diffusion-based feature-distillation paradigm,…
Peer Reviews
Decision·Submitted to ICLR 2026
* The proposed approach of projecting onto otrhogonal directions using the singular vectors of Jacobian is a reasonable and elegant idea. It is also a novel contribution within the context of diffusion-based representation learning for enhancing class-discriminative information. * The observed performance improvement when applying the method to knowledge distillation demonstrates its practical applicability.
* It would be beneficial to provide a clearer methodological or theoretical discussion on why CanoReps work effectively for feature distillation. It remains somewhat unclear why CanoReps sould be preferred over other possible representation extraction methods. For example, how would performance differ if alternative representations were used? In addition to quantiative results, qualitative analyses could further support the claimed effectiveness of CanoReps. * CanoReps are randmoly selected per
1. The problem of extracting robust and interpretable representations in conditional diffusion models is important and timely. 2. The proposed pipeline is conceptually sound: it first identifies canonical latent representations that retain essential categorical information while removing non-discriminative signals, and then leverages these representations to guide the training of a student model with improved adversarial robustness and generalization. 3. Experiments on real-world image dataset
1. The writing and structure of the manuscript could be improved. While the technical content is substantial, the dense presentation and frequent deferrals to the Appendix (e.g., Lines 254–269, 308–320) make the paper difficult to follow. A clearer exposition of the key steps in the main text would improve readability. 2. Experimental settings are insufficiently described. For instance, it is unclear how the “20 different ImageNet classes” (Line 259) were selected. Providing a rationale for thi
- The method introduces a simple yet effective idea: by utilizing the right singular matrix of the Jacobian, it identifies directions that cause significant local variation on the manifold and removes them, thereby isolating class-common representations. The approach demonstrates improved adversarial robustness and resilience to distribution shift, showing consistent performance gains over baselines on both CIFAR-10 and ImageNet datasets. - In addition, this paper demonstrates that the propose
- Numerous hyperparameters: The overall framework appears to be sensitive to hyperparameter choices. For instance, the process of obtaining Canonical Representations (CanoReps) depends on parameters such as $k$, $l$, and $t_{e}$; moreover, CaDistill involves additional hyperparameters ($\lambda_{align}, \lambda_{cano}, \lambda_{dist}$) during training. This raises concerns about the practical applicability and generalizability of the proposed method, as the optimal combination of hyperparameters
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Domain Adaptation and Few-Shot Learning · Face recognition and analysis
MethodsDiffusion