On the Convergence of Momentum-Based Algorithms for Federated Bilevel Optimization Problems

TL;DR

This paper introduces two momentum-based algorithms for federated bilevel optimization, achieving the first linear speedup with respect to device count, supported by theoretical convergence analysis and extensive experiments.

Contribution

The paper develops novel momentum-based algorithms with proven convergence rates that scale linearly with the number of devices in federated bilevel optimization.

Findings

Algorithms achieve linear speedup with device number

Convergence rates established for federated bilevel problems

Experimental results confirm effectiveness

Abstract

In this paper, we studied the federated bilevel optimization problem, which has widespread applications in machine learning. In particular, we developed two momentum-based algorithms for optimizing this kind of problem and established the convergence rate of our two algorithms, providing the sample and communication complexities. Importantly, to the best of our knowledge, our convergence rate is the first one achieving the linear speedup with respect to the number of devices for federated bilevel optimization algorithms. At last, our extensive experimental results confirm the effectiveness of our two algorithms.