Dataset Distillation by Automatic Training Trajectories

TL;DR

This paper introduces Automatic Training Trajectories (ATT), a novel dataset distillation method that adaptively adjusts training trajectory length to improve generalization and stability across different neural network architectures.

Contribution

The paper proposes ATT, an adaptive approach to dataset distillation that dynamically adjusts training trajectory length to mitigate overfitting and improve cross-architecture performance.

Findings

Outperforms existing methods in cross-architecture tests

Enhances stability against parameter variations

Addresses the Accumulated Mismatching Problem (AMP)

Abstract

Dataset Distillation is used to create a concise, yet informative, synthetic dataset that can replace the original dataset for training purposes. Some leading methods in this domain prioritize long-range matching, involving the unrolling of training trajectories with a fixed number of steps (NS) on the synthetic dataset to align with various expert training trajectories. However, traditional long-range matching methods possess an overfitting-like problem, the fixed step size NS forces synthetic dataset to distortedly conform seen expert training trajectories, resulting in a loss of generality-especially to those from unencountered architecture. We refer to this as the Accumulated Mismatching Problem (AMP), and propose a new approach, Automatic Training Trajectories (ATT), which dynamically and adaptively adjusts trajectory length NS to address the AMP. Our method outperforms existing methods particularly in tests involving cross-architectures. Moreover, owing to its adaptive nature, it exhibits enhanced stability in the face of parameter variations.