Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach

TL;DR

This paper introduces GradSamp, a Gaussian sampling method that reduces energy consumption in deep learning training by approximating gradients and enabling epoch skipping, validated across various models and tasks.

Contribution

GradSamp is a novel gradient approximation technique leveraging Gaussian sampling to improve training efficiency and energy savings in deep learning models.

Findings

Achieves significant energy reduction without performance loss.

Effective across CNNs, transformers, and diverse tasks.

Applicable in out-of-distribution and decentralized scenarios.

Abstract

Computing the loss gradient via backpropagation consumes considerable energy during deep learning (DL) model training. In this paper, we propose a novel approach to efficiently compute DL models' gradients to mitigate the substantial energy overhead associated with backpropagation. Exploiting the over-parameterized nature of DL models and the smoothness of their loss landscapes, we propose a method called {\em GradSamp} for sampling gradient updates from a Gaussian distribution. Specifically, we update model parameters at a given epoch (chosen periodically or randomly) by perturbing the parameters (element-wise) from the previous epoch with Gaussian ``noise''. The parameters of the Gaussian distribution are estimated using the error between the model parameter values from the two previous epochs. {\em GradSamp} not only streamlines gradient computation but also enables skipping entire epochs, thereby enhancing overall efficiency. We rigorously validate our hypothesis across a diverse set of standard and non-standard CNN and transformer-based models, spanning various computer vision tasks such as image classification, object detection, and image segmentation. Additionally, we explore its efficacy in out-of-distribution scenarios such as Domain Adaptation (DA), Domain Generalization (DG), and decentralized settings like Federated Learning (FL). Our experimental results affirm the effectiveness of {\em GradSamp} in achieving notable energy savings without compromising performance, underscoring its versatility and potential impact in practical DL applications.