HyperPower: Power- and Memory-Constrained Hyper-Parameter Optimization for Neural Networks

TL;DR

HyperPower introduces a novel framework for efficient hyper-parameter optimization of neural networks under power and memory constraints, utilizing predictive models and Bayesian methods to significantly accelerate the process and improve accuracy.

Contribution

It is the first to use power as a known constraint and develop predictive models for power and memory of neural networks on GPUs, enhancing optimization efficiency.

Findings

Achieves up to 112.99x faster hyper-parameter tuning

Reaches up to 30.12x better test error with constraints

Speeds up evaluations by up to 57.20x

Abstract

While selecting the hyper-parameters of Neural Networks (NNs) has been so far treated as an art, the emergence of more complex, deeper architectures poses increasingly more challenges to designers and Machine Learning (ML) practitioners, especially when power and memory constraints need to be considered. In this work, we propose HyperPower, a framework that enables efficient Bayesian optimization and random search in the context of power- and memory-constrained hyper-parameter optimization for NNs running on a given hardware platform. HyperPower is the first work (i) to show that power consumption can be used as a low-cost, a priori known constraint, and (ii) to propose predictive models for the power and memory of NNs executing on GPUs. Thanks to HyperPower, the number of function evaluations and the best test error achieved by a constraint-unaware method are reached up to 112.99x and 30.12x faster, respectively, while never considering invalid configurations. HyperPower significantly speeds up the hyper-parameter optimization, achieving up to 57.20x more function evaluations compared to constraint-unaware methods for a given time interval, effectively yielding significant accuracy improvements by up to 67.6%.