A Fast and Convergent Proximal Algorithm for Regularized Nonconvex and Nonsmooth Bi-level Optimization

TL;DR

This paper introduces a fast, convergent proximal gradient algorithm for nonconvex, nonsmooth bi-level optimization problems in machine learning, improving computational efficiency and convergence guarantees.

Contribution

It proposes a novel proximal gradient algorithm with Nesterov's momentum for nonconvex bi-level problems, providing rigorous convergence analysis and improved complexity bounds.

Findings

Algorithm converges to a critical point of the bi-level problem.

Achieves an improved complexity of $ ext{O}(\kappa^{3.5}\epsilon^{-2})$.

Effective in hyper-parameter optimization experiments.

Abstract

Many important machine learning applications involve regularized nonconvex bi-level optimization. However, the existing gradient-based bi-level optimization algorithms cannot handle nonconvex or nonsmooth regularizers, and they suffer from a high computation complexity in nonconvex bi-level optimization. In this work, we study a proximal gradient-type algorithm that adopts the approximate implicit differentiation (AID) scheme for nonconvex bi-level optimization with possibly nonconvex and nonsmooth regularizers. In particular, the algorithm applies the Nesterov's momentum to accelerate the computation of the implicit gradient involved in AID. We provide a comprehensive analysis of the global convergence properties of this algorithm through identifying its intrinsic potential function. In particular, we formally establish the convergence of the model parameters to a critical point of the bi-level problem, and obtain an improved computation complexity $\mathcal{O}(\kappa^{3.5}\epsilon^{-2})$ over the state-of-the-art result. Moreover, we analyze the asymptotic convergence rates of this algorithm under a class of local nonconvex geometries characterized by a {\L}ojasiewicz-type gradient inequality. Experiment on hyper-parameter optimization demonstrates the effectiveness of our algorithm.