Theoretical aspects in penalty hyperparameters optimization

TL;DR

This paper explores theoretical foundations for tuning penalty hyperparameters in unsupervised learning, proposing conditions for minimizer existence and an iterative algorithm with a stopping rule based on variational principles.

Contribution

It introduces a bi-level formulation for hyperparameter tuning in unsupervised learning and provides theoretical conditions for minimizer existence in infinite-dimensional spaces.

Findings

Conditions for the existence of a minimizer are outlined.

An iterative algorithm with a variational stopping criterion is proposed.

Theoretical insights applicable when exact minimizers are unattainable.

Abstract

Learning processes are useful methodologies able to improve knowledge of real phenomena. These are often dependent on hyperparameters, variables set before the training process and regulating the learning procedure. Hyperparameters optimization problem is an open issue in learning approaches since it can strongly affect any real data analysis. They are usually selected using Grid-Search or Cross Validation techniques. No automatic tuning procedure exists especially if we focus on an unsupervised learning scenario. This study aims to assess some theoretical considerations for tuning penalty hyperparameters in optimization problems. It considers a bi-level formulation tuning problem in an unsupervised context, by using Gradient-based methods. Suitable conditions for the existence of a minimizer in an infinite-dimensional Hilbert space are outlined, together with some theoretical results, applicable in all those situations when it is unnecessary or not possible obtaining an exact minimizer. An iterative algorithmic strategy is considered, equipped with a stopping criterion via Ekeland's variational principle.