DecHW: Heterogeneous Decentralized Federated Learning Exploiting Second-Order Information

TL;DR

DecHW introduces a second-order information-based aggregation method for decentralized federated learning, effectively handling data and model heterogeneity to improve convergence and reduce communication costs.

Contribution

The paper proposes a novel second-order information-based aggregation approach that explicitly addresses heterogeneity in decentralized federated learning.

Findings

Achieves robust model aggregation across heterogeneous devices

Demonstrates improved convergence with reduced communication overhead

Shows strong generalizability in computer vision tasks

Abstract

Decentralized Federated Learning (DFL) is a serverless collaborative machine learning paradigm where devices collaborate directly with neighbouring devices to exchange model information for learning a generalized model. However, variations in individual experiences and different levels of device interactions lead to data and model initialization heterogeneities across devices. Such heterogeneities leave variations in local model parameters across devices that leads to slower convergence. This paper tackles the data and model heterogeneity by explicitly addressing the parameter level varying evidential credence across local models. A novel aggregation approach is introduced that captures these parameter variations in local models and performs robust aggregation of neighbourhood local updates. Specifically, consensus weights are generated via approximation of second-order information of local models on their local datasets. These weights are utilized to scale neighbourhood updates before aggregating them into global neighbourhood representation. In extensive experiments with computer vision tasks, the proposed approach shows strong generalizability of local models at reduced communication costs.