Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

TL;DR

This paper introduces a novel dataset distillation method for pre-trained vision models that leverages a closed-form solution for linear probing, significantly reducing computational costs while maintaining high performance.

Contribution

It proposes CLP-DD, a bilevel formulation that exploits the closed-form solution for linear probes, improving efficiency and effectiveness over existing trajectory-based methods.

Findings

CLP-DD outperforms LGM without DSA on ImageNet-100.

On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA.

CLP-DD is roughly 14 times faster and uses less than one-eighth of GPU memory.

Abstract

Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14\times$ faster and using less than one-eighth of the GPU memory.