A Framework for Bilevel Optimization on Riemannian Manifolds

TL;DR

This paper introduces a comprehensive framework for bilevel optimization on Riemannian manifolds, including hypergradient estimation, convergence analysis, stochastic extensions, and practical applications.

Contribution

It presents novel hypergradient estimation strategies and convergence analysis for bilevel optimization on Riemannian manifolds, extending to stochastic settings and general retractions.

Findings

Effective hypergradient estimation methods on manifolds

Convergence guarantees for the proposed algorithms

Successful application to real-world problems

Abstract

Bilevel optimization has gained prominence in various applications. In this study, we introduce a framework for solving bilevel optimization problems, where the variables in both the lower and upper levels are constrained on Riemannian manifolds. We present several hypergradient estimation strategies on manifolds and analyze their estimation errors. Furthermore, we provide comprehensive convergence and complexity analyses for the proposed hypergradient descent algorithm on manifolds. We also extend our framework to encompass stochastic bilevel optimization and incorporate the use of general retraction. The efficacy of the proposed framework is demonstrated through several applications.