Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimization

TL;DR

CYBER-0 is a novel zero-order federated learning algorithm that achieves high communication and memory efficiency while being resilient to Byzantine faults, with proven convergence guarantees.

Contribution

It introduces CYBER-0, the first zero-order method for Byzantine-resilient federated learning with theoretical convergence guarantees.

Findings

Outperforms state-of-the-art algorithms in communication efficiency

Achieves similar accuracy to existing methods on MNIST and RoBERTa-Large

Provides theoretical convergence guarantees for convex loss functions

Abstract

We introduce CYBER-0, the first zero-order optimization algorithm for memory-and-communication efficient Federated Learning, resilient to Byzantine faults. We show through extensive numerical experiments on the MNIST dataset and finetuning RoBERTa-Large that CYBER-0 outperforms state-of-the-art algorithms in terms of communication and memory efficiency while reaching similar accuracy. We provide theoretical guarantees on its convergence for convex loss functions.