Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning

TL;DR

Far-HO is a software package built on TensorFlow that unifies hyperparameter optimization and meta-learning through bilevel programming, enabling automatic tuning of learning rates, example weighting, and hyper-representations.

Contribution

It introduces a practical implementation of bilevel programming for hyperparameter optimization and meta-learning, facilitating seamless application to deep learning tasks.

Findings

Efficient optimization of learning rates and loss weights.

Unified framework for hyperparameter optimization and meta-learning.

Open-source package compatible with TensorFlow.

Abstract

In (Franceschi et al., 2018) we proposed a unified mathematical framework, grounded on bilevel programming, that encompasses gradient-based hyperparameter optimization and meta-learning. We formulated an approximate version of the problem where the inner objective is solved iteratively, and gave sufficient conditions ensuring convergence to the exact problem. In this work we show how to optimize learning rates, automatically weight the loss of single examples and learn hyper-representations with Far-HO, a software package based on the popular deep learning framework TensorFlow that allows to seamlessly tackle both HO and ML problems.