Decentralized Learning of Generative Adversarial Networks from Non-iid Data

TL;DR

This paper introduces a novel decentralized GAN training method called F2U and F2A, enabling learning from non-iid data across clients while preserving data privacy, with proven theoretical guarantees and empirical success.

Contribution

It proposes the Forgiver-First Update (F2U) and Forgiver-First Aggregation (F2A) methods for decentralized GAN training on non-iid data, with theoretical analysis and practical effectiveness.

Findings

F2U achieves global optimum distribution including all classes.

F2A adaptively aggregates discriminators, improving practical performance.

Empirical results outperform state-of-the-art decentralized GAN methods.

Abstract

This work addresses a new problem that learns generative adversarial networks (GANs) from multiple data collections that are each i) owned separately by different clients and ii) drawn from a non-identical distribution that comprises different classes. Given such non-iid data as input, we aim to learn a distribution involving all the classes input data can belong to, while keeping the data decentralized in each client storage. Our key contribution to this end is a new decentralized approach for learning GANs from non-iid data called Forgiver-First Update (F2U), which a) asks clients to train an individual discriminator with their own data and b) updates a generator to fool the most `forgiving' discriminators who deem generated samples as the most real. Our theoretical analysis proves that this updating strategy allows the decentralized GAN to achieve a generator's distribution with all the input classes as its global optimum based on f-divergence minimization. Moreover, we propose a relaxed version of F2U called Forgiver-First Aggregation (F2A) that performs well in practice, which adaptively aggregates the discriminators while emphasizing forgiving ones. Our empirical evaluations with image generation tasks demonstrated the effectiveness of our approach over state-of-the-art decentralized learning methods.