MOFA: Modular Factorial Design for Hyperparameter Optimization

TL;DR

MOFA is a lightweight hyperparameter optimization method that combines factorial design exploration with factorial analysis exploitation, improving efficiency and effectiveness over existing methods.

Contribution

Introduces MOFA, a novel HPO approach that alternates exploration and exploitation using factorial design and analysis, with proven higher confidence and parallelizable rounds.

Findings

MOFA outperforms state-of-the-art HPO methods in effectiveness.

MOFA offers significant efficiency improvements due to parallelizable rounds.

Empirical results validate MOFA's superior performance across benchmarks.

Abstract

This paper presents a novel and lightweight hyperparameter optimization (HPO) method, MOdular FActorial Design (MOFA). MOFA pursues several rounds of HPO, where each round alternates between exploration of hyperparameter space by factorial design and exploitation of evaluation results by factorial analysis. Each round first explores the configuration space by constructing a low-discrepancy set of hyperparameters that cover this space well while de-correlating hyperparameters, and then exploits evaluation results through factorial analysis that determines which hyperparameters should be further explored and which should become fixed in the next round. We prove that the inference of MOFA achieves higher confidence than other sampling schemes. Each individual round is highly parallelizable and hence offers major improvements of efficiency compared to model-based methods. Empirical results show that MOFA achieves better effectiveness and efficiency compared with state-of-the-art methods.