ACHO: Adaptive Conformal Hyperparameter Optimization

TL;DR

This paper introduces ACHO, a flexible hyperparameter optimization framework using conformal confidence intervals, which outperforms random search in tuning random forests and CNNs by making fewer assumptions.

Contribution

It presents a novel conformal-based hyperparameter search method that relaxes distributional assumptions, allowing for more versatile search model architectures.

Findings

ACHO achieves satisfactory interval coverage.

ACHO demonstrates superior tuning performance.

Outperforms random search in benchmarks.

Abstract

Several novel frameworks for hyperparameter search have emerged in the last decade, but most rely on strict, often normal, distributional assumptions, limiting search model flexibility. This paper proposes a novel optimization framework based on upper confidence bound sampling of conformal confidence intervals, whose weaker assumption of exchangeability enables greater choice of search model architectures. Several such architectures were explored and benchmarked on hyperparameter search of random forests and convolutional neural networks, displaying satisfactory interval coverage and superior tuning performance to random search.