The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset

TL;DR

This paper explores how various training parameters and mechanisms affect decentralized federated learning performance on the MNIST dataset, highlighting robustness and failure modes in different setups.

Contribution

It introduces Decentralized Federated Learning (DFL) and systematically evaluates its robustness under various training modifications using the MNIST dataset.

Findings

Training procedures are generally robust but not optimal.

Large variance in model weights can cause training failures.

Partial sharing and non-IID data impact model convergence.

Abstract

Federated Learning is an algorithm suited for training models on decentralized data, but the requirement of a central "server" node is a bottleneck. In this document, we first introduce the notion of Decentralized Federated Learning (DFL). We then perform various experiments on different setups, such as changing model aggregation frequency, switching from independent and identically distributed (IID) dataset partitioning to non-IID partitioning with partial global sharing, using different optimization methods across clients, and breaking models into segments with partial sharing. All experiments are run on the MNIST handwritten digits dataset. We observe that those altered training procedures are generally robust, albeit non-optimal. We also observe failures in training when the variance between model weights is too large. The open-source experiment code is accessible through GitHub\footnote{Code was uploaded at \url{https://github.com/zhzhang2018/DecentralizedFL}}.