Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selection

TL;DR

This paper introduces a diffusion-prior-based patch selection method for dataset distillation, enabling efficient one-step process and improved performance on large-scale datasets like ImageNet-1K.

Contribution

It proposes a novel patch selection framework using diffusion priors, addressing distribution shift issues and reducing the need for multiple distillation steps.

Findings

Outperforms state-of-the-art methods across various metrics

Enables one-step dataset distillation process

Effective on large-scale datasets like ImageNet-1K

Abstract

Dataset distillation provides an effective approach to reduce memory and computational costs by optimizing a compact dataset that achieves performance comparable to the full original. However, for large-scale datasets and complex deep networks (e.g., ImageNet-1K with ResNet-101), the vast optimization space hinders distillation effectiveness, limiting practical applications. Recent methods leverage pre-trained diffusion models to directly generate informative images, thereby bypassing pixel-level optimization and achieving promising results. Nonetheless, these approaches often suffer from distribution shifts between the pre-trained diffusion prior and target datasets, as well as the need for multiple distillation steps under varying settings. To overcome these challenges, we propose a novel framework that is orthogonal to existing diffusion-based distillation techniques by utilizing the diffusion prior for patch selection rather than generation. Our method predicts noise from the diffusion model conditioned on input images and optional text prompts (with or without label information), and computes the associated loss for each image-patch pair. Based on the loss differences, we identify distinctive regions within the original images. Furthermore, we apply intra-class clustering and ranking on the selected patches to enforce diversity constraints. This streamlined pipeline enables a one-step distillation process. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art methods across various metrics and settings.