MetaDD: Boosting Dataset Distillation with Neural Network Architecture-Invariant Generalization

TL;DR

MetaDD enhances dataset distillation by extracting architecture-invariant features, significantly improving cross-architecture neural network performance and generalizability across various methods and datasets.

Contribution

MetaDD introduces a novel approach to improve dataset distillation's generalization across neural network architectures by partitioning data into meta and heterogeneous features and using an architecture-invariant loss.

Findings

MetaDD achieves up to 30.1% accuracy on Distilled Tiny-Imagenet.

MetaDD outperforms existing methods like GLaD by 1.7%.

MetaDD can be integrated into any dataset distillation method with low memory overhead.

Abstract

Dataset distillation (DD) entails creating a refined, compact distilled dataset from a large-scale dataset to facilitate efficient training. A significant challenge in DD is the dependency between the distilled dataset and the neural network (NN) architecture used. Training a different NN architecture with a distilled dataset distilled using a specific architecture often results in diminished trainning performance for other architectures. This paper introduces MetaDD, designed to enhance the generalizability of DD across various NN architectures. Specifically, MetaDD partitions distilled data into meta features (i.e., the data's common characteristics that remain consistent across different NN architectures) and heterogeneous features (i.e., the data's unique feature to each NN architecture). Then, MetaDD employs an architecture-invariant loss function for multi-architecture feature alignment, which increases meta features and reduces heterogeneous features in distilled data. As a low-memory consumption component, MetaDD can be seamlessly integrated into any DD methodology. Experimental results demonstrate that MetaDD significantly improves performance across various DD methods. On the Distilled Tiny-Imagenet with Sre2L (50 IPC), MetaDD achieves cross-architecture NN accuracy of up to 30.1\%, surpassing the second-best method (GLaD) by 1.7\%.