Improving Noise Efficiency in Privacy-preserving Dataset Distillation

TL;DR

This paper proposes a new method for privacy-preserving dataset distillation that improves efficiency and accuracy by decoupling sampling and optimization and reducing the impact of differential privacy noise, leading to better synthetic datasets.

Contribution

The authors introduce a novel framework that enhances private dataset distillation by decoupling sampling from optimization and mitigating DP noise through subspace matching, improving convergence and signal quality.

Findings

Achieves 10.0% improvement on CIFAR-10 with 50 images per class.

Increases performance by 8.3% with one-fifth the dataset size of previous methods.

Demonstrates significant advancement in privacy-preserving dataset distillation.

Abstract

Modern machine learning models heavily rely on large datasets that often include sensitive and private information, raising serious privacy concerns. Differentially private (DP) data generation offers a solution by creating synthetic datasets that limit the leakage of private information within a predefined privacy budget; however, it requires a substantial amount of data to achieve performance comparable to models trained on the original data. To mitigate the significant expense incurred with synthetic data generation, Dataset Distillation (DD) stands out for its remarkable training and storage efficiency. This efficiency is particularly advantageous when integrated with DP mechanisms, curating compact yet informative synthetic datasets without compromising privacy. However, current state-of-the-art private DD methods suffer from a synchronized sampling-optimization process and the dependency on noisy training signals from randomly initialized networks. This results in the inefficient utilization of private information due to the addition of excessive noise. To address these issues, we introduce a novel framework that decouples sampling from optimization for better convergence and improves signal quality by mitigating the impact of DP noise through matching in an informative subspace. On CIFAR-10, our method achieves a \textbf{10.0\%} improvement with 50 images per class and \textbf{8.3\%} increase with just \textbf{one-fifth} the distilled set size of previous state-of-the-art methods, demonstrating significant potential to advance privacy-preserving DD.