DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning

TL;DR

DP-KFC introduces a data-free method for curvature estimation in differentially private deep learning, improving optimization without privacy or public data, especially useful in sensitive domains.

Contribution

It proposes DP-KFC, a novel approach to construct KFAC preconditioners using synthetic noise, eliminating the need for private or public data in privacy-preserving deep learning.

Findings

DP-KFC outperforms DP-SGD and adaptive baselines across modalities.

It matches private-data preconditioners without privacy budget consumption.

Public-data variants degrade performance, highlighting the effectiveness of data-free estimation.

Abstract

Differentially private optimization suffers from a fundamental geometric mismatch: deep networks have highly anisotropic loss landscapes, yet DP-SGD injects isotropic noise. Second-order preconditioning can resolve this, but estimating curvature typically requires private data (consuming privacy budget) or public data (introducing distribution shift). We show that the Fisher Information Matrix decouples into architectural sensitivity, recoverable via synthetic noise, and input correlations, approximable from modality-specific frequency statistics. We propose DP-KFC, which constructs KFAC preconditioners by probing networks with structured synthetic noise, requiring neither private nor public data. Empirically, DP-KFC consistently outperforms DP-SGD and adaptive baselines across diverse modalities in strong privacy regimes ($\varepsilon \leq 3$). DP-KFC matches private-data preconditioners while public-data variants degrade by up to $4.8\%$, showing that curvature can be estimated without consuming privacy budget or introducing distribution shift. This enables privacy-preserving learning in specialized domains (e.g., medical applications) where regulatory constraints make data scarce.