Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks

TL;DR

This paper introduces difficulty-guided sampling (DGS) to improve dataset distillation by aligning the distilled data with the difficulty levels relevant to downstream image classification tasks, enhancing performance.

Contribution

It proposes a novel difficulty-guided sampling method and difficulty-aware guidance to better bridge the gap between distillation objectives and downstream task requirements.

Findings

DGS improves downstream classification accuracy.

Difficulty-aware guidance enhances dataset quality.

Method shows broad potential for various tasks.

Abstract

In this paper, we propose difficulty-guided sampling (DGS) to bridge the target gap between the distillation objective and the downstream task, therefore improving the performance of dataset distillation. Deep neural networks achieve remarkable performance but have time and storage-consuming training processes. Dataset distillation is proposed to generate compact, high-quality distilled datasets, enabling effective model training while maintaining downstream performance. Existing approaches typically focus on features extracted from the original dataset, overlooking task-specific information, which leads to a target gap between the distillation objective and the downstream task. We propose leveraging characteristics that benefit the downstream training into data distillation to bridge this gap. Focusing on the downstream task of image classification, we introduce the concept of difficulty and propose DGS as a plug-in post-stage sampling module. Following the specific target difficulty distribution, the final distilled dataset is sampled from image pools generated by existing methods. We also propose difficulty-aware guidance (DAG) to explore the effect of difficulty in the generation process. Extensive experiments across multiple settings demonstrate the effectiveness of the proposed methods. It also highlights the broader potential of difficulty for diverse downstream tasks.