A Survey on Green Deep Learning

TL;DR

This survey reviews recent advances in green deep learning, emphasizing energy-efficient models and strategies to reduce carbon footprint while maintaining performance across AI applications.

Contribution

It systematically categorizes and discusses progress and challenges in green deep learning technologies, including compact networks and energy-efficient training and inference methods.

Findings

Progress in compact network architectures

Development of energy-efficient training strategies

Identification of unresolved challenges in green AI

Abstract

In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and computer vision (CV). However, despite promising results, it needs to be noted that the computations required by SOTA models have been increased at an exponential rate. Massive computations not only have a surprisingly large carbon footprint but also have negative effects on research inclusiveness and deployment on real-world applications. Green deep learning is an increasingly hot research field that appeals to researchers to pay attention to energy usage and carbon emission during model training and inference. The target is to yield novel results with lightweight and efficient technologies. Many technologies can be used to achieve this goal, like model compression and knowledge distillation. This paper focuses on presenting a systematic review of the development of Green deep learning technologies. We classify these approaches into four categories: (1) compact networks, (2) energy-efficient training strategies, (3) energy-efficient inference approaches, and (4) efficient data usage. For each category, we discuss the progress that has been achieved and the unresolved challenges.