Robust and Efficient Aggregation for Distributed Learning

TL;DR

This paper introduces new robust and statistically efficient aggregation methods for distributed learning, improving resilience to malicious agents while maintaining high sample efficiency compared to existing robust schemes.

Contribution

The paper proposes novel aggregation algorithms that are both robust to outliers and more sample-efficient than existing median-based methods.

Findings

Achieves robustness against malicious agents.

Maintains high sample efficiency in non-contaminated settings.

Outperforms existing robust aggregation schemes in experiments.

Abstract

Distributed learning paradigms, such as federated and decentralized learning, allow for the coordination of models across a collection of agents, and without the need to exchange raw data. Instead, agents compute model updates locally based on their available data, and subsequently share the update model with a parameter server or their peers. This is followed by an aggregation step, which traditionally takes the form of a (weighted) average. Distributed learning schemes based on averaging are known to be susceptible to outliers. A single malicious agent is able to drive an averaging-based distributed learning algorithm to an arbitrarily poor model. This has motivated the development of robust aggregation schemes, which are based on variations of the median and trimmed mean. While such procedures ensure robustness to outliers and malicious behavior, they come at the cost of significantly reduced sample efficiency. This means that current robust aggregation schemes require significantly higher agent participation rates to achieve a given level of performance than their mean-based counterparts in non-contaminated settings. In this work we remedy this drawback by developing statistically efficient and robust aggregation schemes for distributed learning.