Training Diffusion Models with Federated Learning

TL;DR

This paper introduces a federated learning approach for training diffusion models that preserves data privacy, reduces communication costs, and maintains high image quality, addressing privacy and transparency concerns in AI model training.

Contribution

The paper presents a novel federated diffusion model training scheme that adapts FedAvg for diffusion models, significantly reducing communication overhead while preserving image quality.

Findings

Achieved up to 74% reduction in exchanged parameters.

Maintained image quality comparable to centralized training.

Demonstrated effective collaborative training without exposing local data.

Abstract

The training of diffusion-based models for image generation is predominantly controlled by a select few Big Tech companies, raising concerns about privacy, copyright, and data authority due to their lack of transparency regarding training data. To ad-dress this issue, we propose a federated diffusion model scheme that enables the independent and collaborative training of diffusion models without exposing local data. Our approach adapts the Federated Averaging (FedAvg) algorithm to train a Denoising Diffusion Model (DDPM). Through a novel utilization of the underlying UNet backbone, we achieve a significant reduction of up to 74% in the number of parameters exchanged during training,compared to the naive FedAvg approach, whilst simultaneously maintaining image quality comparable to the centralized setting, as evaluated by the FID score.