Energy Considerations for Large Pretrained Neural Networks

TL;DR

This paper evaluates how different compression techniques for large neural networks impact their electricity consumption, revealing that steganographic capacity reduction significantly reduces energy use compared to pruning and low-rank factorization.

Contribution

It provides a comparative analysis of energy consumption for various compression methods on large pre-trained models, highlighting the effectiveness of steganographic capacity reduction.

Findings

Steganographic capacity reduction greatly decreases energy consumption.

Pruning and low-rank factorization show minimal energy savings.

Compression techniques vary significantly in their impact on electricity use.

Abstract

Increasingly complex neural network architectures have achieved phenomenal performance. However, these complex models require massive computational resources that consume substantial amounts of electricity, which highlights the potential environmental impact of such models. Previous studies have demonstrated that substantial redundancies exist in large pre-trained models. However, previous work has primarily focused on compressing models while retaining comparable model performance, and the direct impact on electricity consumption appears to have received relatively little attention. By quantifying the energy usage associated with both uncompressed and compressed models, we investigate compression as a means of reducing electricity consumption. We consider nine different pre-trained models, ranging in size from 8M parameters to 138M parameters. To establish a baseline, we first train each model without compression and record the electricity usage and time required during training, along with other relevant statistics. We then apply three compression techniques: Steganographic capacity reduction, pruning, and low-rank factorization. In each of the resulting cases, we again measure the electricity usage, training time, model accuracy, and so on. We find that pruning and low-rank factorization offer no significant improvements with respect to energy usage or other related statistics, while steganographic capacity reduction provides major benefits in almost every case. We discuss the significance of these findings.