Automated Computational Energy Minimization of ML Algorithms using Constrained Bayesian Optimization

TL;DR

This paper introduces a constrained Bayesian optimization approach to minimize energy consumption in machine learning model training while maintaining high predictive performance, addressing the growing concern of energy costs.

Contribution

The paper presents a novel application of constrained Bayesian optimization to reduce energy use in ML training without sacrificing accuracy.

Findings

CBO reduces energy consumption compared to traditional methods.

Predictive performance remains high despite energy minimization.

Effective on both regression and classification tasks.

Abstract

Bayesian optimization (BO) is an efficient framework for optimization of black-box objectives when function evaluations are costly and gradient information is not easily accessible. BO has been successfully applied to automate the task of hyperparameter optimization (HPO) in machine learning (ML) models with the primary objective of optimizing predictive performance on held-out data. In recent years, however, with ever-growing model sizes, the energy cost associated with model training has become an important factor for ML applications. Here we evaluate Constrained Bayesian Optimization (CBO) with the primary objective of minimizing energy consumption and subject to the constraint that the generalization performance is above some threshold. We evaluate our approach on regression and classification tasks and demonstrate that CBO achieves lower energy consumption without compromising the predictive performance of ML models.