Optimizing Hyperparameters with Conformal Quantile Regression

TL;DR

This paper introduces a robust hyperparameter optimization method using conformalized quantile regression, which better models uncertainty and accelerates convergence compared to traditional Gaussian process-based approaches.

Contribution

It proposes a novel HPO approach leveraging conformalized quantile regression and a multi-fidelity aggregation technique, improving robustness and efficiency.

Findings

Faster convergence on empirical benchmarks

Outperforms traditional Gaussian process methods

Effective multi-fidelity resource aggregation

Abstract

Many state-of-the-art hyperparameter optimization (HPO) algorithms rely on model-based optimizers that learn surrogate models of the target function to guide the search. Gaussian processes are the de facto surrogate model due to their ability to capture uncertainty but they make strong assumptions about the observation noise, which might not be warranted in practice. In this work, we propose to leverage conformalized quantile regression which makes minimal assumptions about the observation noise and, as a result, models the target function in a more realistic and robust fashion which translates to quicker HPO convergence on empirical benchmarks. To apply our method in a multi-fidelity setting, we propose a simple, yet effective, technique that aggregates observed results across different resource levels and outperforms conventional methods across many empirical tasks.