Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios

TL;DR

This paper introduces EDF, a dataset distillation method that emphasizes discriminative features using Grad-CAM, significantly improving performance in complex scenarios like ImageNet-1K subsets.

Contribution

EDF is a novel dataset distillation approach that enhances key discriminative regions in synthetic images and introduces the Comp-DD benchmark for complex scenarios.

Findings

EDF outperforms state-of-the-art methods in complex scenarios

The approach effectively emphasizes discriminative features using Grad-CAM

The new benchmark facilitates research in complex dataset distillation

Abstract

Dataset distillation has demonstrated strong performance on simple datasets like CIFAR, MNIST, and TinyImageNet but struggles to achieve similar results in more complex scenarios. In this paper, we propose EDF (emphasizes the discriminative features), a dataset distillation method that enhances key discriminative regions in synthetic images using Grad-CAM activation maps. Our approach is inspired by a key observation: in simple datasets, high-activation areas typically occupy most of the image, whereas in complex scenarios, the size of these areas is much smaller. Unlike previous methods that treat all pixels equally when synthesizing images, EDF uses Grad-CAM activation maps to enhance high-activation areas. From a supervision perspective, we downplay supervision signals that have lower losses, as they contain common patterns. Additionally, to help the DD community better explore complex scenarios, we build the Complex Dataset Distillation (Comp-DD) benchmark by meticulously selecting sixteen subsets, eight easy and eight hard, from ImageNet-1K. In particular, EDF consistently outperforms SOTA results in complex scenarios, such as ImageNet-1K subsets. Hopefully, more researchers will be inspired and encouraged to improve the practicality and efficacy of DD. Our code and benchmark will be made public at https://github.com/NUS-HPC-AI-Lab/EDF.