An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation

TL;DR

This paper introduces an adaptive inexact method for bilevel learning that efficiently computes hypergradients using primal-dual differentiation, with error bounds and adaptive step-size strategies to improve optimization accuracy and efficiency.

Contribution

It develops an inexact bilevel optimization approach with error estimation and adaptive step-size, enhancing the efficiency of hypergradient computation in bilevel learning.

Findings

Effective hypergradient estimation with error bounds

Adaptive step-size improves convergence

Application to learned regularizers like input-convex neural networks

Abstract

We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network.