Minimizing Energy Consumption of Deep Learning Models by Energy-Aware Training

TL;DR

This paper introduces EAT, a gradient-based training algorithm that reduces energy consumption of deep learning models by incorporating a differentiable sparsity penalty, achieving better energy-performance trade-offs.

Contribution

The paper presents a novel energy-aware training method using a differentiable approximation of the norm to promote sparsity and reduce energy use during training.

Findings

EAT reduces energy consumption while maintaining classification accuracy.

Experimental results on three datasets show improved energy-efficiency trade-offs.

EAT outperforms baseline training methods in energy efficiency.

Abstract

Deep learning models undergo a significant increase in the number of parameters they possess, leading to the execution of a larger number of operations during inference. This expansion significantly contributes to higher energy consumption and prediction latency. In this work, we propose EAT, a gradient-based algorithm that aims to reduce energy consumption during model training. To this end, we leverage a differentiable approximation of the $\ell_0$ norm, and use it as a sparse penalty over the training loss. Through our experimental analysis conducted on three datasets and two deep neural networks, we demonstrate that our energy-aware training algorithm EAT is able to train networks with a better trade-off between classification performance and energy efficiency.