Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization

TL;DR

This paper introduces a quantile-based learn-then-test method for hyperparameter optimization that offers statistical guarantees on risk measures, improving reliability in engineering AI applications.

Contribution

It extends the learn-then-test calibration framework to control quantiles of risk measures, providing robust statistical guarantees for hyperparameter tuning.

Findings

Demonstrates the method's effectiveness on a radio access scheduling problem.

Provides statistical guarantees on quantiles of risk measures.

Enhances robustness of AI model calibration in engineering contexts.

Abstract

The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.