Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation

TL;DR

This paper addresses the challenge of bias in long-tailed dataset distillation by analyzing soft-label biases and proposing ADSA, a calibration module that improves tail-class accuracy and overall performance.

Contribution

It introduces an imbalance-aware generalization bound and a novel Adaptive Soft-label Alignment (ADSA) module to mitigate soft-label bias in long-tailed dataset distillation.

Findings

ADSA improves tail-class accuracy by up to 11.8%.

Overall accuracy on ImageNet-1k-LT reaches 41.4%.

ADSA enhances performance across various distillation techniques.

Abstract

Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation. Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels. To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8% and raises overall accuracy to 41.4%. Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques. Code is available at: https://github.com/j-cyoung/ADSA_DD.git.