Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matching

TL;DR

Teddy introduces a Taylor-approximated dataset distillation method that significantly improves efficiency and performance for large-scale datasets like ImageNet by reducing computational complexity and leveraging pre-cached models.

Contribution

The paper proposes a novel Taylor-based approximation and a pre-cached model pool to enhance large-scale dataset distillation efficiency and effectiveness.

Findings

Achieves state-of-the-art efficiency and performance on ImageNet datasets.

Reduces runtime by 46.6% compared to prior methods.

Surpasses previous methods by up to 12.8% in performance.

Abstract

Dataset distillation or condensation refers to compressing a large-scale dataset into a much smaller one, enabling models trained on this synthetic dataset to generalize effectively on real data. Tackling this challenge, as defined, relies on a bi-level optimization algorithm: a novel model is trained in each iteration within a nested loop, with gradients propagated through an unrolled computation graph. However, this approach incurs high memory and time complexity, posing difficulties in scaling up to large datasets such as ImageNet. Addressing these concerns, this paper introduces Teddy, a Taylor-approximated dataset distillation framework designed to handle large-scale dataset and enhance efficiency. On the one hand, backed up by theoretical analysis, we propose a memory-efficient approximation derived from Taylor expansion, which transforms the original form dependent on multi-step gradients to a first-order one. On the other hand, rather than repeatedly training a novel model in each iteration, we unveil that employing a pre-cached pool of weak models, which can be generated from a single base model, enhances both time efficiency and performance concurrently, particularly when dealing with large-scale datasets. Extensive experiments demonstrate that the proposed Teddy attains state-of-the-art efficiency and performance on the Tiny-ImageNet and original-sized ImageNet-1K dataset, notably surpassing prior methods by up to 12.8%, while reducing 46.6% runtime. Our code will be available at https://github.com/Lexie-YU/Teddy.