A Weighted Loss Approach to Robust Federated Learning under Data Heterogeneity

TL;DR

This paper introduces WoLA, a weighted loss function that improves Byzantine-resilient federated learning in heterogeneous data environments by aligning honest gradients and effectively identifying malicious ones.

Contribution

The paper proposes the Worker Label Alignment Loss (WoLA), a novel weighted loss that enhances Byzantine resilience in federated learning under data heterogeneity.

Findings

WoLA outperforms existing methods in heterogeneous settings.

Theoretical analysis supports WoLA's effectiveness.

Empirical results demonstrate improved robustness against Byzantine attacks.

Abstract

Federated learning (FL) is a machine learning paradigm that enables multiple data holders to collaboratively train a machine learning model without sharing their training data with external parties. In this paradigm, workers locally update a model and share with a central server their updated gradients (or model parameters). While FL seems appealing from a privacy perspective, it opens a number of threats from a security perspective as (Byzantine) participants can contribute poisonous gradients (or model parameters) harming model convergence. Byzantine-resilient FL addresses this issue by ensuring that the training proceeds as if Byzantine participants were absent. Towards this purpose, common strategies ignore outlier gradients during model aggregation, assuming that Byzantine gradients deviate more from honest gradients than honest gradients do from each other. However, in heterogeneous settings, honest gradients may differ significantly, making it difficult to distinguish honest outliers from Byzantine ones. In this paper, we introduce the Worker Label Alignement Loss (WoLA), a weighted loss that aligns honest worker gradients despite data heterogeneity, which facilitates the identification of Byzantines' gradients. This approach significantly outperforms state-of-the-art methods in heterogeneous settings. In this paper, we provide both theoretical insights and empirical evidence of its effectiveness.