Conditional Gradient Methods with Standard LMO for Stochastic Simple Bilevel Optimization

TL;DR

This paper introduces an efficient conditional gradient method for stochastic simple bilevel optimization that avoids costly projections, achieves favorable convergence rates, and demonstrates practical benefits in regression and dictionary learning tasks.

Contribution

It develops a novel iteratively regularized conditional gradient approach using only linear optimization oracles, with proven convergence rates for stochastic and finite-sum settings.

Findings

Achieves $O(t^{-1/4})$ convergence in stochastic convex setting.

Achieves $O(t^{-1/2})$ convergence in finite-sum convex setting.

Demonstrates practical improvements in regression and dictionary learning experiments.

Abstract

We propose efficient methods for solving stochastic simple bilevel optimization problems with convex inner levels, where the goal is to minimize an outer stochastic objective function subject to the solution set of an inner stochastic optimization problem. Existing methods often rely on costly projection or linear optimization oracles over complex sets, limiting their scalability. To overcome this, we propose an iteratively regularized conditional gradient approach that leverages linear optimization oracles exclusively over the base feasible set. Our proposed methods employ a vanishing regularization sequence that progressively emphasizes the inner problem while biasing towards desirable minimal outer objective solutions. In the one-sample stochastic setting and under standard convexity assumptions, we establish non-asymptotic convergence rates of $O(t^{-1/4})$ for both the outer and inner objectives. In the finite-sum setting with a mini-batch scheme, the corresponding rates become $O(t^{-1/2})$. When the outer objective is nonconvex, we prove non-asymptotic convergence rates of $O(t^{-1/7})$ for both the outer and inner objectives in the one-sample stochastic setting, and $O(t^{-1/4})$ in the finite-sum setting. Experimental results on over-parametrized regression and dictionary learning tasks demonstrate the practical advantages of our approach over existing methods, confirming our theoretical findings.