Communication-Efficient Federated Bilevel Optimization with Local and Global Lower Level Problems

TL;DR

This paper introduces FedBiOAcc, a communication-efficient algorithm for federated bilevel optimization that achieves linear speedup and demonstrates superior empirical performance in real-world tasks.

Contribution

It proposes FedBiOAcc, the first communication-efficient federated bilevel optimization algorithm with accelerated hyper-gradient estimation and convergence guarantees.

Findings

Achieves $O(rac{1}{ ext{epsilon}})$ communication complexity.

Demonstrates linear speedup with the number of clients.

Outperforms existing methods in federated data-cleaning and hyper-representation learning.

Abstract

Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact of Federated Learning's inherent challenges on the convergence of bilevel algorithms remain obscure. In this work, we investigate Federated Bilevel Optimization problems and propose a communication-efficient algorithm, named FedBiOAcc. The algorithm leverages an efficient estimation of the hyper-gradient in the distributed setting and utilizes the momentum-based variance-reduction acceleration. Remarkably, FedBiOAcc achieves a communication complexity $O(\epsilon^{-1})$, a sample complexity $O(\epsilon^{-1.5})$ and the linear speed up with respect to the number of clients. We also analyze a special case of the Federated Bilevel Optimization problems, where lower level problems are locally managed by clients. We prove that FedBiOAcc-Local, a modified version of FedBiOAcc, converges at the same rate for this type of problems. Finally, we validate the proposed algorithms through two real-world tasks: Federated Data-cleaning and Federated Hyper-representation Learning. Empirical results show superior performance of our algorithms.