Neural Spectral Decomposition for Dataset Distillation

TL;DR

This paper introduces Neural Spectrum Decomposition, a novel dataset distillation method that models datasets as low-rank high-dimensional observations, achieving state-of-the-art results on multiple benchmarks.

Contribution

It presents a new low-rank spectrum tensor framework for dataset distillation, enabling efficient data reconstruction and sharing, with trajectory matching optimization.

Findings

Achieves state-of-the-art performance on CIFAR10, CIFAR100, Tiny ImageNet, and ImageNet Subset.

Introduces spectrum tensor and transformation matrix approach for dataset representation.

Demonstrates efficient data sharing and reconstruction in distillation process.

Abstract

In this paper, we propose Neural Spectrum Decomposition, a generic decomposition framework for dataset distillation. Unlike previous methods, we consider the entire dataset as a high-dimensional observation that is low-rank across all dimensions. We aim to discover the low-rank representation of the entire dataset and perform distillation efficiently. Toward this end, we learn a set of spectrum tensors and transformation matrices, which, through simple matrix multiplication, reconstruct the data distribution. Specifically, a spectrum tensor can be mapped back to the image space by a transformation matrix, and efficient information sharing during the distillation learning process is achieved through pairwise combinations of different spectrum vectors and transformation matrices. Furthermore, we integrate a trajectory matching optimization method guided by a real distribution. Our experimental results demonstrate that our approach achieves state-of-the-art performance on benchmarks, including CIFAR10, CIFAR100, Tiny Imagenet, and ImageNet Subset. Our code are available at \url{https://github.com/slyang2021/NSD}.