Byzantine-Resilient Zero-Order Optimization for Communication-Efficient Heterogeneous Federated Learning

TL;DR

CyBeR-0 is a novel Byzantine-resilient federated zero-order optimization method that ensures robustness against attacks, reduces communication costs, and maintains stable performance across diverse learning tasks.

Contribution

The paper introduces CyBeR-0, a new method combining Byzantine resilience with zero-order optimization and transformed robust aggregation for heterogeneous federated learning.

Findings

Achieves robustness against Byzantine attacks

Reduces communication costs significantly

Maintains stable performance on large language models

Abstract

We introduce CyBeR-0, a Byzantine-resilient federated zero-order optimization method that is robust under Byzantine attacks and provides significant savings in uplink and downlink communication costs. We introduce transformed robust aggregation to give convergence guarantees for general non-convex objectives under client data heterogeneity. Empirical evaluations for standard learning tasks and fine-tuning large language models show that CyBeR-0 exhibits stable performance with only a few scalars per-round communication cost and reduced memory requirements.