Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling

TL;DR

This paper proposes a novel framework for long-tailed dataset distillation that addresses bias and statistical estimation issues by using unbiased recovery and relabeling techniques, significantly improving performance on imbalanced datasets.

Contribution

The paper introduces a unified level framework with three components for unbiased recovery and relabeling, addressing limitations of trajectory-based methods in long-tailed dataset distillation.

Findings

Achieves 15.6% top-1 accuracy improvement on CIFAR-100-LT

Achieves 11.8% top-1 accuracy improvement on Tiny-ImageNet-LT

Consistently outperforms state-of-the-art methods across various class imbalance levels

Abstract

Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance. Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10. Codes are available at https://github.com/2018cx/RLDD.