Gradient Inversion of Federated Diffusion Models

TL;DR

This paper demonstrates that gradient inversion attacks can effectively reconstruct high-resolution images from federated diffusion model training, revealing significant privacy risks despite privacy-preserving measures.

Contribution

The paper introduces GIDM, a novel two-phase optimization method leveraging generative models as priors for image reconstruction from gradients, and extends it to GIDM+ to handle private training noise and sampling steps.

Findings

Gradient inversion can nearly perfectly reconstruct original images.

Sharing gradients in federated diffusion models poses serious privacy risks.

High-resolution images are vulnerable to inversion attacks even with privacy measures.

Abstract

Diffusion models are becoming defector generative models, which generate exceptionally high-resolution image data. Training effective diffusion models require massive real data, which is privately owned by distributed parties. Each data party can collaboratively train diffusion models in a federated learning manner by sharing gradients instead of the raw data. In this paper, we study the privacy leakage risk of gradient inversion attacks. First, we design a two-phase fusion optimization, GIDM, to leverage the well-trained generative model itself as prior knowledge to constrain the inversion search (latent) space, followed by pixel-wise fine-tuning. GIDM is shown to be able to reconstruct images almost identical to the original ones. Considering a more privacy-preserving training scenario, we then argue that locally initialized private training noise $\epsilon$ and sampling step t may raise additional challenges for the inversion attack. To solve this, we propose a triple-optimization GIDM+ that coordinates the optimization of the unknown data, $\epsilon$ and $t$. Our extensive evaluation results demonstrate the vulnerability of sharing gradient for data protection of diffusion models, even high-resolution images can be reconstructed with high quality.