Distributed Bilevel Optimization with Dual Pruning for Resource-limited Clients

TL;DR

This paper introduces a resource-adaptive distributed bilevel optimization framework enabling low-resource clients to efficiently optimize models by reducing computation and providing theoretical convergence guarantees.

Contribution

It proposes the first resource-adaptive bilevel optimization framework with a second-order free hypergradient estimator for resource-limited clients.

Findings

Achieves an asymptotically optimal convergence rate of O(1/√(C_x*Q)).

Demonstrates effectiveness and efficiency through extensive experiments.

Supports resource-limited clients in distributed bilevel optimization.

Abstract

With the development of large-scale models, traditional distributed bilevel optimization algorithms cannot be applied directly in low-resource clients. The key reason lies in the excessive computation involved in optimizing both the lower- and upper-level functions. Thus, we present the first resource-adaptive distributed bilevel optimization framework with a second-order free hypergradient estimator, which allows each client to optimize the submodels adapted to the available resources. Due to the coupled influence of partial outer parameters x and inner parameters y, it's challenging to theoretically analyze the upper bound regarding the globally averaged hypergradient for full model parameters. The error bound of inner parameter also needs to be reformulated since the local partial training. The provable theorems show that both RABO and RAFBO can achieve an asymptotically optimal convergence rate of $O(1/\sqrt{C_x^{\ast}Q})$, which is dominated by the minimum coverage of the outer parameter $C_x^{\ast}$. Extensive experiments on two different tasks demonstrate the effectiveness and computation efficiency of our proposed methods.