Can Adjusting Hyperparameters Lead to Green Deep Learning: An Empirical Study on Correlations between Hyperparameters and Energy Consumption of Deep Learning Models

TL;DR

This study empirically investigates how hyperparameter adjustments affect the energy consumption of deep learning models, revealing that proper tuning can reduce energy use without sacrificing performance, especially in parallel training scenarios.

Contribution

It introduces a mutation-based approach to analyze hyperparameter impacts on energy consumption and highlights the potential for greener deep learning through hyperparameter tuning.

Findings

Hyperparameters show significant correlation with energy consumption.

Adjusting hyperparameters can reduce energy use without performance loss.

Energy consumption in parallel training is more sensitive to hyperparameter changes.

Abstract

Context: Along with developing Deep learning (DL) models, larger datasets and more complex model structures are applied, leading to rising computing resources and energy consumption, which is an alert that green DL models should receive more attention. Objective: This paper focuses on a novel view to analyze DL energy consumption: the effect of hyperparameters on the energy cost of DL models. Method: Our approach involves using mutation operators to simulate how practitioners adjust hyperparameters, such as epochs and learning rates. We train the original and mutated models separately and gather energy information and run-time performance metrics. Moreover, we focus on the parallel scenario where multiple DL models are trained in parallel. Results: To examine the effect of hyperparameters on energy consumption, we conducted extensive experiments on five real-world DL models. The results show that (1) many hyperparameters studied have a (positive or negative) correlation with energy consumption, (2) adjusting hyperparameters can make DL models greener, i.e., lead to less energy consumption without performance damage, and (3) in a parallel environment, energy consumption becomes more susceptible to change. Conclusions: We suggest that hyperparameters need more attention in developing DL models, as appropriately adjusting hyperparameters would cause green DL models.