PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces

TL;DR

PED-ANOVA introduces a novel, efficient method for quantifying hyperparameter importance in arbitrary subspaces, aiding hyperparameter space design for deep learning models.

Contribution

It develops a new formulation of f-ANOVA for arbitrary subspaces and proposes PED-ANOVA, a computationally efficient algorithm using Pearson divergence for hyperparameter importance estimation.

Findings

Successfully identifies important hyperparameters in various subspaces

Demonstrates high computational efficiency

Outperforms existing methods in relevance and speed

Abstract

The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models. In turn, designing a good HP space is critically dependent on understanding the role of different HPs. This motivates research on HP Importance (HPI), e.g., with the popular method of functional ANOVA (f-ANOVA). However, the original f-ANOVA formulation is inapplicable to the subspaces most relevant to algorithm designers, such as those defined by top performance. To overcome this issue, we derive a novel formulation of f-ANOVA for arbitrary subspaces and propose an algorithm that uses Pearson divergence (PED) to enable a closed-form calculation of HPI. We demonstrate that this new algorithm, dubbed PED-ANOVA, is able to successfully identify important HPs in different subspaces while also being extremely computationally efficient.