Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

TL;DR

This paper introduces condPED-ANOVA, a new framework for accurately estimating hyperparameter importance in conditional search spaces, addressing limitations of previous methods that assumed unconditional spaces.

Contribution

It develops a closed-form estimator for hyperparameter importance that accounts for conditional activation and domain changes, improving interpretability in complex search spaces.

Findings

Naive adaptations of existing estimators are misleading in conditional settings.

condPED-ANOVA provides meaningful and interpretable hyperparameter importance estimates.

The method is validated through experiments demonstrating its effectiveness.

Abstract

We propose conditional PED-ANOVA (condPED-ANOVA), a principled framework for estimating hyperparameter importance (HPI) in conditional search spaces, where the presence or domain of a hyperparameter can depend on other hyperparameters. Although the original PED-ANOVA provides a fast and efficient way to estimate HPI within the top-performing regions of the search space, it assumes a fixed, unconditional search space and therefore cannot properly handle conditional hyperparameters. To address this, we introduce a conditional HPI for top-performing regions and derive a closed-form estimator that accurately reflects conditional activation and domain changes. Experiments show that naive adaptations of existing HPI estimators yield misleading or uninterpretable importances in conditional settings, whereas condPED-ANOVA consistently provides meaningful importances that reflect the underlying conditional structure. Our code is publicly available at https://github.com/kAIto47802/condPED-ANOVA.