Interim Report on Human-Guided Adaptive Hyperparameter Optimization with Multi-Fidelity Sprints

TL;DR

This paper presents a human-guided, multi-fidelity hyperparameter optimization method using phased Bayesian sessions, hyperband pruning, and meta-learning to efficiently tune NLP model variants.

Contribution

It introduces a phased, multi-fidelity hyperparameter optimization approach with human guidance and meta-learning for NLP models.

Findings

Efficient hyperparameter space pruning with low-fidelity sprints.

Improved model tuning using multi-fidelity and hyperband strategies.

Meta-learner effectively tunes classification thresholds.

Abstract

This case study applies a phased hyperparameter optimization process to compare multitask natural language model variants that utilize multiphase learning rate scheduling and optimizer parameter grouping. We employ short, Bayesian optimization sessions that leverage multi-fidelity, hyperparameter space pruning, progressive halving, and a degree of human guidance. We utilize the Optuna TPE sampler and Hyperband pruner, as well as the Scikit-Learn Gaussian process minimization. Initially, we use efficient low-fidelity sprints to prune the hyperparameter space. Subsequent sprints progressively increase their model fidelity and employ hyperband pruning for efficiency. A second aspect of our approach is using a meta-learner to tune threshold values to resolve classification probabilities during inference. We demonstrate our method on a collection of variants of the 2021 Joint Entity and Relation Extraction model proposed by Eberts and Ulges.