Ensuring Reliability via Hyperparameter Selection: Review and Advances

TL;DR

This paper reviews the Learn-Then-Test framework for hyperparameter selection in AI models, highlighting recent advances that provide statistical guarantees and extensions for practical engineering scenarios.

Contribution

It offers a comprehensive review of the LTT framework and introduces new extensions for risk measures, multi-objective optimization, prior knowledge, and adaptivity in hyperparameter selection.

Findings

Statistical guarantees can be achieved in hyperparameter selection.

Extensions improve applicability to engineering scenarios.

Illustrative applications demonstrate practical relevance.

Abstract

Hyperparameter selection is a critical step in the deployment of artificial intelligence (AI) models, particularly in the current era of foundational, pre-trained, models. By framing hyperparameter selection as a multiple hypothesis testing problem, recent research has shown that it is possible to provide statistical guarantees on population risk measures attained by the selected hyperparameter. This paper reviews the Learn-Then-Test (LTT) framework, which formalizes this approach, and explores several extensions tailored to engineering-relevant scenarios. These extensions encompass different risk measures and statistical guarantees, multi-objective optimization, the incorporation of prior knowledge and dependency structures into the hyperparameter selection process, as well as adaptivity. The paper also includes illustrative applications for communication systems.