Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?

TL;DR

This paper introduces C²R, a novel robust dataset distillation framework that improves adversarial robustness by focusing on small-margin adversaries and increasing class separation.

Contribution

C²R couples an attack-aware curriculum with a contrastive robustness loss to enhance robustness and boundary separation in dataset distillation.

Findings

C²R outperforms prior methods by 2.8% in robust accuracy on multiple datasets.

The method effectively prioritizes small-margin adversaries for improved robustness.

Experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet subsets validate the approach.

Abstract

Dataset distillation (DD) compresses a large training set into a small synthetic set for efficient training, but most DD methods optimize only clean accuracy and leave robustness uncontrolled. Recent robust DD methods improve robustness, yet they often suffer from a poor accuracy-robustness trade-off because they (i) treat all adversarially perturbed examples uniformly, despite robust risk being dominated by near-zero robust margins, and (ii) do not explicitly increase inter-class separation in the decision boundary where attacks concentrate. We present Contrastive Curriculum for Robust Dataset Distillation (C$^2$R), a framework that couples an attack-aware curriculum with a contrastive robustness objective. From a robust-margin perspective, we derive a perturbation score that approximates each sample's robust hinge, enabling a curriculum that prioritizes the smallest-margin adversaries that most directly drive robust error. In parallel, a class-balanced contrastive robustness loss enforces adversarial invariance while explicitly widening boundary separation across classes. Experiments on CIFAR-10/100, Tiny-ImageNet, and multiple ImageNet-1K subsets under six attacks show that C$^2$R achieves the best robust accuracy, outperforming prior robust DD by $2.8$% on average.