TL;DR
CONCORD introduces a concept-informed diffusion approach leveraging large language models to improve dataset distillation by enhancing control and detail accuracy, achieving state-of-the-art results on ImageNet-1K.
Contribution
This paper presents a novel concept-informed diffusion method that incorporates LLM-derived concepts to improve dataset distillation's controllability and detail fidelity.
Findings
Achieves state-of-the-art performance on ImageNet-1K.
Enhances controllability and interpretability of distilled datasets.
No reliance on pre-trained classifiers.
Abstract
Dataset distillation (DD) has witnessed significant progress in creating small datasets that encapsulate rich information from large original ones. Particularly, methods based on generative priors show promising performance, while maintaining computational efficiency and cross-architecture generalization. However, the generation process lacks explicit controllability for each sample. Previous distillation methods primarily match the real distribution from the perspective of the entire dataset, whereas overlooking concept completeness at the instance level. The missing or incorrectly represented object details cannot be efficiently compensated due to the constrained sample amount typical in DD settings. To this end, we propose incorporating the concept understanding of large language models (LLMs) to perform Concept-Informed Diffusion (CONCORD) for dataset distillation. Specifically,…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
- The proposed method utilizes diffusion models and CLIP to generate condensed images without the need for additional model training, streamlining the process and saving computational resources. - The method successfully automates the concept extraction process through the use of large language models (LLMs), enhancing efficiency and reducing the reliance on manual intervention. - The paper demonstrates reliable performance gains when integrated as an add-on to existing methods, showcasing its v
- The proposed approach primarily leverages backpropagation within the CLIP feature space, which gives the impression of being a minimal extension of existing diffusion-based methods with CLIP feature matching. Rather than integrating concept-informed insights, it appears to be focused on distilling CLIP's knowledge directly. - The method demonstrates unstable performance on IPC 1, raising concerns about its scalability to larger datasets or its generalizability across different dataset sizes wi
- This method demonstrates significant innovation, being the first to combine LLMs with Dataset Distillation (DD). It leverages the vast knowledge base of LLMs and guides the dataset distillation process through conceptual knowledge. - The method greatly enhances instance-level control, addressing the issue of insufficient detail control in existing approaches to a certain extent. - By using CLIP to verify the correlation between concepts and images, the method ensures the validity and accurac
- The experimental results are highly dependent on the concept information provided by LLMs, which may lead to instability in performance. - The performance improvements are limited, and there is a lack of detailed comparison with other methods in cross-architecture evaluations. - The introduction of contrastive matching and concept evaluation may increase computational costs.
1. The proposed method is simple and easy to implement, making it easily integrable into existing generative-model-based data distillation approaches. Additionally, the use of large language models (LLMs) is straightforward and compatible with more advanced LLMs. 2. The experimental results show consistent improvements across multiple benchmark datasets. While the method does not outperform state-of-the-art techniques, it enhances the performance of baseline methods in nearly all cases.
1. The method appears heuristic, combining LLMs with diffusion models. Specifically, the concept-informed diffusion is based on the classifier-guided diffusion model, where the formulation is derived from conditional probabilities. However, this paper directly alters how conditional information is incorporated, replacing classifier guidance with concept information (the gradient of the loss function), which seems questionable. Providing explanations or justifications for these modifications in t
The paper's visualizations clearly demonstrate improved image detail when using CONCORD. The overall idea is innovative, as it utilizes LLM-generated content as prompts to guide the diffusion process during dataset distillation, showcasing novelty. CONCORD allows for explicit control during the diffusion process, offering a more interpretable approach compared to traditional dataset distillation methods. The theoretical foundations are well-established, and the inclusion of code is a pleasant
The performance improvements are quite limited, with some of the gains potentially attributable to variance. This method depends on the quality of concepts retrieved from LLMs; if the descriptions are not sufficiently accurate or detailed, the quality of the generated datasets could suffer. The introduction of LLMs and contrastive loss increases the complexity of training.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsDiffusion
