Spend More to Save More (SM2): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization

TL;DR

This paper introduces SM2, an energy-aware hyperparameter optimization method based on successive halving that reduces energy consumption while maintaining model performance, addressing sustainability in machine learning model tuning.

Contribution

SM2 is a novel energy-efficient hyperparameter optimization approach that incorporates hardware and energy tracking into successive halving, improving sustainability in model development.

Findings

SM2 reduces energy consumption during hyperparameter tuning.

SM2 maintains comparable model performance to traditional methods.

SM2 effectively prevents energy waste across diverse datasets and hardware.

Abstract

A fundamental step in the development of machine learning models commonly involves the tuning of hyperparameters, often leading to multiple model training runs to work out the best-performing configuration. As machine learning tasks and models grow in complexity, there is an escalating need for solutions that not only improve performance but also address sustainability concerns. Existing strategies predominantly focus on maximizing the performance of the model without considering energy efficiency. To bridge this gap, in this paper, we introduce Spend More to Save More (SM2), an energy-aware hyperparameter optimization implementation based on the widely adopted successive halving algorithm. Unlike conventional approaches including energy-intensive testing of individual hyperparameter configurations, SM2 employs exploratory pretraining to identify inefficient configurations with minimal energy expenditure. Incorporating hardware characteristics and real-time energy consumption tracking, SM2 identifies an optimal configuration that not only maximizes the performance of the model but also enables energy-efficient training. Experimental validations across various datasets, models, and hardware setups confirm the efficacy of SM2 to prevent the waste of energy during the training of hyperparameter configurations.