Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation

TL;DR

This paper introduces RETA, a novel dataset distillation framework combining dynamic patch retrieval and persistent homology regularization to improve synthetic data quality and generalization across multiple datasets.

Contribution

RETA integrates dynamic retrieval and topology alignment to address static patch limitations in dataset distillation, enhancing diversity and performance.

Findings

RETA outperforms baselines on CIFAR-100, Tiny-ImageNet, and ImageNet-1K.

Achieves 64.3% top-1 accuracy on ImageNet-1K with 50 images per class.

Consistently improves generalization and efficiency in dataset distillation.

Abstract

Decoupled dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA -- a Retrieval and Topology Alignment framework for decoupled DD. First, Dynamic Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.