Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

TL;DR

This paper introduces HALD, a new training paradigm that combines hard and soft labels to mitigate local semantic drift caused by limited soft-label supervision, improving generalization in dataset distillation and classification tasks.

Contribution

It theoretically analyzes semantic drift under soft labels and proposes a hybrid approach, HALD, that effectively integrates hard labels to enhance training stability and accuracy.

Findings

HALD improves generalization on ImageNet-1K by 9.0% over prior methods.

Hybridizing hard and soft labels reduces semantic drift and distribution misalignment.

Extensive experiments validate the effectiveness of the proposed approach.

Abstract

Soft labels generated by teacher models have become a dominant paradigm for knowledge transfer and recent large-scale dataset distillation such as SRe2L, RDED, LPLD, offering richer supervision than conventional hard labels. However, we observe that when only a limited number of crops per image are used, soft labels are prone to local semantic drift: a crop may visually resemble another class, causing its soft embedding to deviate from the ground-truth semantics of the original image. This mismatch between local visual content and global semantic meaning introduces systematic errors and distribution misalignment between training and testing. In this work, we revisit the overlooked role of hard labels and show that, when appropriately integrated, they provide a powerful content-agnostic anchor to calibrate semantic drift. We theoretically characterize the emergence of drift under few soft-label supervision and demonstrate that hybridizing soft and hard labels restores alignment between visual content and semantic supervision. Building on this insight, we propose a new training paradigm, Hard Label for Alleviating Local Semantic Drift (HALD), which leverages hard labels as intermediate corrective signals while retaining the fine-grained advantages of soft labels. Extensive experiments on dataset distillation and large-scale conventional classification benchmarks validate our approach, showing consistent improvements in generalization. On ImageNet-1K, we achieve 42.7% with only 285M storage for soft labels, outperforming prior state-of-the-art LPLD by 9.0%. Our findings re-establish the importance of hard labels as a complementary tool, and call for a rethinking of their role in soft-label-dominated training.