Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillation

TL;DR

This paper introduces Hierarchical Parameterization Distillation (H-PD), a novel dataset distillation method that explores hierarchical features to improve synthetic dataset quality and performance under high compression ratios.

Contribution

The paper proposes a hierarchical feature exploration approach for dataset distillation, addressing limitations of fixed optimization spaces and introducing a new feature distance metric.

Findings

H-PD outperforms existing methods at high compression ratios.

H-PD surpasses generative distillation with diffusion models under extreme compression.

The hierarchical optimization strategy improves synthetic dataset quality.

Abstract

Dataset distillation is an emerging dataset reduction method, which condenses large-scale datasets while maintaining task accuracy. Current parameterization methods achieve enhanced performance under extremely high compression ratio by optimizing determined synthetic dataset in informative feature domain. However, they limit themselves to a fixed optimization space for distillation, neglecting the diverse guidance across different informative latent spaces. To overcome this limitation, we propose a novel parameterization method dubbed Hierarchical Parameterization Distillation (H-PD), to systematically explore hierarchical feature within provided feature space (e.g., layers within pre-trained generative adversarial networks). We verify the correctness of our insights by applying the hierarchical optimization strategy on GAN-based parameterization method. In addition, we introduce a novel class-relevant feature distance metric to alleviate the computational burden associated with synthetic dataset evaluation, bridging the gap between synthetic and original datasets. Experimental results demonstrate that the proposed H-PD achieves a significant performance improvement under various settings with equivalent time consumption, and even surpasses current generative distillation using diffusion models under extreme compression ratios IPC=1 and IPC=10.