Trust-Aware Diversion for Data-Effective Distillation

TL;DR

This paper introduces a Trust-Aware Diversion (TAD) method for dataset distillation that effectively handles mislabeled samples by iteratively distinguishing trusted data from untrusted, improving distillation performance in noisy label scenarios.

Contribution

The paper proposes a novel dual-loop optimization framework that dynamically separates trusted and untrusted samples, enhancing dataset distillation robustness against label noise.

Findings

Significant performance improvements on CIFAR10, CIFAR100, and Tiny ImageNet.

Effective handling of symmetric, asymmetric, and real-world label noise.

Robust distillation results in challenging mislabeled settings.

Abstract

Dataset distillation compresses a large dataset into a small synthetic subset that retains essential information. Existing methods assume that all samples are perfectly labeled, limiting their real-world applications where incorrect labels are ubiquitous. These mislabeled samples introduce untrustworthy information into the dataset, which misleads model optimization in dataset distillation. To tackle this issue, we propose a Trust-Aware Diversion (TAD) dataset distillation method. Our proposed TAD introduces an iterative dual-loop optimization framework for data-effective distillation. Specifically, the outer loop divides data into trusted and untrusted spaces, redirecting distillation toward trusted samples to guarantee trust in the distillation process. This step minimizes the impact of mislabeled samples on dataset distillation. The inner loop maximizes the distillation objective by recalibrating untrusted samples, thus transforming them into valuable ones for distillation. This dual-loop iteratively refines and compensates for each other, gradually expanding the trusted space and shrinking the untrusted space. Experiments demonstrate that our method can significantly improve the performance of existing dataset distillation methods on three widely used benchmarks (CIFAR10, CIFAR100, and Tiny ImageNet) in three challenging mislabeled settings (symmetric, asymmetric, and real-world).