Generalized Kernel Inducing Points by Duality Gap for Dataset Distillation

TL;DR

DGKIP extends Kernel Inducing Points for dataset distillation by using duality gap theory, enabling support for various loss functions and providing theoretical guarantees, while maintaining efficiency and strong empirical results.

Contribution

DGKIP introduces a duality gap-based approach to dataset distillation, removing the need for bi-level optimization and supporting diverse loss functions.

Findings

Effective on MNIST and CIFAR-10 benchmarks.

Supports multiple loss functions beyond squared loss.

Provides theoretical bounds on test error and prediction consistency.

Abstract

We propose Duality Gap KIP (DGKIP), an extension of the Kernel Inducing Points (KIP) method for dataset distillation. While existing dataset distillation methods often rely on bi-level optimization, DGKIP eliminates the need for such optimization by leveraging duality theory in convex programming. The KIP method has been introduced as a way to avoid bi-level optimization; however, it is limited to the squared loss and does not support other loss functions (e.g., cross-entropy or hinge loss) that are more suitable for classification tasks. DGKIP addresses this limitation by exploiting an upper bound on parameter changes after dataset distillation using the duality gap, enabling its application to a wider range of loss functions. We also characterize theoretical properties of DGKIP by providing upper bounds on the test error and prediction consistency after dataset distillation. Experimental results on standard benchmarks such as MNIST and CIFAR-10 demonstrate that DGKIP retains the efficiency of KIP while offering broader applicability and robust performance.