Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data

TL;DR

FedRCO is a new second-order optimization method for federated learning that enhances stability, reduces communication costs, and improves convergence in non-IID data environments.

Contribution

It introduces a robust framework with real-time anomaly detection, stability protocols, and adaptive aggregation to address instability and inefficiency in distributed second-order optimization.

Findings

FedRCO outperforms existing methods in convergence speed and accuracy.

It effectively mitigates instability caused by non-IID data.

Experiments demonstrate higher robustness and efficiency in diverse scenarios.

Abstract

In this paper, we present Federated Robust Curvature Optimization (FedRCO), a novel second-order optimization framework designed to improve convergence speed and reduce communication cost in Federated Learning systems under statistical heterogeneity. Existing second-order optimization methods are often computationally expensive and numerically unstable in distributed settings. In contrast, FedRCO addresses these challenges by integrating an efficient approximate curvature optimizer with a provable stability mechanism. Specifically, FedRCO incorporates three key components: (1) a Gradient Anomaly Monitor that detects and mitigates exploding gradients in real-time, (2) a Fail-Safe Resilience protocol that resets optimization states upon numerical instability, and (3) a Curvature-Preserving Adaptive Aggregation strategy that safely integrates global knowledge without erasing the local curvature geometry. Theoretical analysis shows that FedRCO can effectively mitigate instability and prevent unbounded updates while preserving optimization efficiency. Extensive experiments show that FedRCO achieves superior robustness against diverse non-IID scenarios while achieving higher accuracy and faster convergence than both state-of-the-art first-order and second-order methods.