Towards Mitigating Architecture Overfitting on Distilled Datasets

TL;DR

This paper proposes methods like DropPath and knowledge distillation to reduce architecture overfitting in dataset distillation, enabling models to generalize better across different architectures and sizes.

Contribution

The paper introduces novel techniques to mitigate architecture overfitting in dataset distillation, improving generalization across various network architectures and capacities.

Findings

Significantly reduces architecture overfitting across tasks.

Enables training with distilled datasets to generalize to larger test networks.

Achieves comparable or superior performance on larger test architectures.

Abstract

Dataset distillation methods have demonstrated remarkable performance for neural networks trained with very limited training data. However, a significant challenge arises in the form of \textit{architecture overfitting}: the distilled training dataset synthesized by a specific network architecture (i.e., training network) generates poor performance when trained by other network architectures (i.e., test networks), especially when the test networks have a larger capacity than the training network. This paper introduces a series of approaches to mitigate this issue. Among them, DropPath renders the large model to be an implicit ensemble of its sub-networks, and knowledge distillation ensures each sub-network acts similarly to the small but well-performing teacher network. These methods, characterized by their smoothing effects, significantly mitigate architecture overfitting. We conduct extensive experiments to demonstrate the effectiveness and generality of our methods. Particularly, across various scenarios involving different tasks and different sizes of distilled data, our approaches significantly mitigate architecture overfitting. Furthermore, our approaches achieve comparable or even superior performance when the test network is larger than the training network.