Fractional Order Distributed Optimization

TL;DR

This paper introduces fractional order distributed optimization (FrODO), a new framework that uses fractional-order memory to improve convergence speed and stability in distributed machine learning tasks, especially on ill-conditioned problems.

Contribution

The paper presents a theoretically-grounded fractional order framework that guarantees linear convergence in distributed optimization, outperforming existing methods in speed and stability.

Findings

FrODO achieves up to 4x faster convergence on ill-conditioned problems.

FrODO provides 2-3x speedup in federated neural network training.

The approach maintains stability and theoretical convergence guarantees.

Abstract

Distributed optimization is fundamental to modern machine learning applications like federated learning, but existing methods often struggle with ill-conditioned problems and face stability-versus-speed tradeoffs. We introduce fractional order distributed optimization (FrODO); a theoretically-grounded framework that incorporates fractional-order memory terms to enhance convergence properties in challenging optimization landscapes. Our approach achieves provable linear convergence for any strongly connected network. Through empirical validation, our results suggest that FrODO achieves up to 4 times faster convergence versus baselines on ill-conditioned problems and 2-3 times speedup in federated neural network training, while maintaining stability and theoretical guarantees.