ZOBA: An Efficient Single-loop Zeroth-order Bilevel Optimization Algorithm
Marco Rando, Samuel Vaiter

TL;DR
This paper introduces ZOBA, a novel single-loop zeroth-order bilevel optimization algorithm that efficiently approximates hypergradients without nested loops, outperforming prior methods in convergence and computational cost.
Contribution
We propose ZOBA, the first finite-difference single-loop bilevel optimization algorithm that uses delayed information for hypergradient approximation, improving efficiency over existing two-loop methods.
Findings
Achieves convergence with complexity $igo(p(d + p)^2 ext{ extbackslash}varepsilon^{-2})$
Outperforms prior Hessian-based approaches in efficiency
Demonstrates effectiveness on synthetic and real-world tasks
Abstract
Bilevel optimization problems consist of minimizing a value function whose evaluation depends on the solution of an inner optimization problem. These problems are typically tackled using first-order methods that require computing the gradient of the value function ({\it the hypergradient}). In several practical settings, however, first-order information is unavailable ({\it zeroth-order setting}), rendering these methods inapplicable. Finite-difference methods provide an alternative by approximating hypergradients using function evaluations along a set of directions. Nevertheless, such surrogates are notoriously expensive, and existing finite-difference bilevel methods rely on two-loop algorithms that are poorly parallelizable. In this work, we propose ZOBA, the first finite-difference single-loop algorithm for bilevel optimization. Our method leverages finite-difference hypergradient…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStochastic Gradient Optimization Techniques · Sparse and Compressive Sensing Techniques · Optimization and Variational Analysis
