A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization

TL;DR

This paper introduces a near-optimal algorithm for bilevel empirical risk minimization, improving sample complexity bounds and establishing a matching lower bound, thus advancing the theoretical understanding of bilevel optimization in machine learning.

Contribution

It proposes a bilevel extension of the SARAH algorithm with optimal sample complexity bounds and provides a matching lower bound, demonstrating the algorithm's optimality.

Findings

The algorithm requires $ ilde{O}((n+m)^{1/2} ext{epsilon}^{-1})$ oracle calls.

The lower bound on oracle calls matches the algorithm's complexity.

The method advances theoretical understanding of bilevel optimization in machine learning.

Abstract

Bilevel optimization problems, which are problems where two optimization problems are nested, have more and more applications in machine learning. In many practical cases, the upper and the lower objectives correspond to empirical risk minimization problems and therefore have a sum structure. In this context, we propose a bilevel extension of the celebrated SARAH algorithm. We demonstrate that the algorithm requires $\mathcal{O}((n+m)^{\frac12}\varepsilon^{-1})$ oracle calls to achieve $\varepsilon$-stationarity with $n+m$ the total number of samples, which improves over all previous bilevel algorithms. Moreover, we provide a lower bound on the number of oracle calls required to get an approximate stationary point of the objective function of the bilevel problem. This lower bound is attained by our algorithm, making it optimal in terms of sample complexity.