DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning

TL;DR

This paper introduces DFDG, a novel data-free dual-generator adversarial distillation method for one-shot federated learning, enabling robust global model training without public datasets or multiple communication rounds.

Contribution

The paper proposes a new data-free dual-generator approach for one-shot federated learning, addressing limitations of existing methods by exploring broader local model spaces and improving global model accuracy.

Findings

DFDG outperforms state-of-the-art baselines in image classification accuracy.

The dual-generator training enhances diversity and transferability of synthetic data.

Extensive experiments validate the effectiveness of DFDG in various settings.

Abstract

Federated Learning (FL) is a distributed machine learning scheme in which clients jointly participate in the collaborative training of a global model by sharing model information rather than their private datasets. In light of concerns associated with communication and privacy, one-shot FL with a single communication round has emerged as a de facto promising solution. However, existing one-shot FL methods either require public datasets, focus on model homogeneous settings, or distill limited knowledge from local models, making it difficult or even impractical to train a robust global model. To address these limitations, we propose a new data-free dual-generator adversarial distillation method (namely DFDG) for one-shot FL, which can explore a broader local models' training space via training dual generators. DFDG is executed in an adversarial manner and comprises two parts: dual-generator training and dual-model distillation. In dual-generator training, we delve into each generator concerning fidelity, transferability and diversity to ensure its utility, and additionally tailor the cross-divergence loss to lessen the overlap of dual generators' output spaces. In dual-model distillation, the trained dual generators work together to provide the training data for updates of the global model. At last, our extensive experiments on various image classification tasks show that DFDG achieves significant performance gains in accuracy compared to SOTA baselines.