Genealogical Population-Based Training for Hyperparameter Optimization

TL;DR

This paper introduces Genealogical Population Based Training (GPBT), a novel hyperparameter optimization method that leverages shared model histories to improve efficiency and accuracy across different models.

Contribution

GPBT exploits genealogical relationships among models to enhance hyperparameter optimization, reducing computational costs and improving accuracy over existing methods.

Findings

Cuts computational cost by 2-3 times

Achieves about 1% accuracy improvement on vision tasks

Reduces result variance significantly

Abstract

HyperParameter Optimization (HPO) aims at finding the best HyperParameters (HPs) of learning models, such as neural networks, in the fastest and most efficient way possible. Most recent HPO algorithms try to optimize HPs regardless of the model that obtained them, assuming that for different models, same HPs will produce very similar results. We break free from this paradigm and propose a new take on preexisting methods that we called Genealogical Population Based Training (GPBT). GPBT, via the shared histories of "genealogically"-related models, exploit the coupling of HPs and models in an efficient way. We experimentally demonstrate that our method cuts down by 2 to 3 times the computational cost required, generally allows a 1% accuracy improvement on computer vision tasks, and reduces the variance of the results by an order of magnitude, compared to the current algorithms. Our method is search-algorithm agnostic so that the inner search routine can be any search algorithm like TPE, GP, CMA or random search.