An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning

TL;DR

This paper introduces an adaptive inexact first-order method with backtracking line search for bilevel optimization, enabling efficient hyperparameter learning without exact function or gradient evaluations, demonstrated on imaging and data science tasks.

Contribution

It presents a novel algorithm that adaptively determines accuracy levels and step sizes in bilevel optimization, improving robustness and efficiency over existing methods.

Findings

Algorithm converges to a stationary point using inexact evaluations.

Demonstrates robustness to initial hyperparameters and step size.

Effective on imaging and data science problems like denoising and logistic regression.

Abstract

Various tasks in data science are modeled utilizing the variational regularization approach, where manually selecting regularization parameters presents a challenge. The difficulty gets exacerbated when employing regularizers involving a large number of hyperparameters. To overcome this challenge, bilevel learning can be employed to learn such parameters from data. However, neither exact function values nor exact gradients with respect to the hyperparameters are attainable, necessitating methods that only rely on inexact evaluation of such quantities. State-of-the-art inexact gradient-based methods a priori select a sequence of the required accuracies and cannot identify an appropriate step size since the Lipschitz constant of the hypergradient is unknown. In this work, we propose an algorithm with backtracking line search that only relies on inexact function evaluations and hypergradients and show convergence to a stationary point. Furthermore, the proposed algorithm determines the required accuracy dynamically rather than manually selected before running it. Our numerical experiments demonstrate the efficiency and feasibility of our approach for hyperparameter estimation on a range of relevant problems in imaging and data science such as total variation and field of experts denoising and multinomial logistic regression. Particularly, the results show that the algorithm is robust to its own hyperparameters such as the initial accuracies and step size.