Semi-Variance Reduction for Fair Federated Learning

TL;DR

This paper introduces SemiVRed, a novel federated learning algorithm inspired by finance risk models, which improves fairness without sacrificing overall system performance, especially in heterogeneous data scenarios.

Contribution

It proposes SemiVRed, a new fair federated learning algorithm based on semi-variance, balancing fairness and overall performance better than existing methods.

Findings

SemiVRed achieves state-of-the-art fairness and accuracy.

It performs well on vision and language datasets with heterogeneous data.

SemiVRed outperforms existing fair FL algorithms in experiments.

Abstract

Ensuring fairness in a Federated Learning (FL) system, i.e., a satisfactory performance for all of the participating diverse clients, is an important and challenging problem. There are multiple fair FL algorithms in the literature, which have been relatively successful in providing fairness. However, these algorithms mostly emphasize on the loss functions of worst-off clients to improve their performance, which often results in the suppression of well-performing ones. As a consequence, they usually sacrifice the system's overall average performance for achieving fairness. Motivated by this and inspired by two well-known risk modeling methods in Finance, Mean-Variance and Mean-Semi-Variance, we propose and study two new fair FL algorithms, Variance Reduction (VRed) and Semi-Variance Reduction (SemiVRed). VRed encourages equality between clients' loss functions by penalizing their variance. In contrast, SemiVRed penalizes the discrepancy of only the worst-off clients' loss functions from the average loss. Through extensive experiments on multiple vision and language datasets, we show that, SemiVRed achieves SoTA performance in scenarios with heterogeneous data distributions and improves both fairness and system overall average performance.