A Parallel Alternative for Energy-Efficient Neural Network Training and Inferencing

TL;DR

This paper proposes phantom parallelism, an energy-efficient alternative to tensor parallelism for training neural networks, reducing energy consumption by about 50% and enabling smaller models to achieve similar performance with fewer resources.

Contribution

Introduction of phantom parallelism as a novel approach to reduce energy consumption in neural network training and inference, with comprehensive theoretical and empirical validation.

Findings

Approximately 50% reduction in energy consumption compared to tensor parallelism

Lower bandwidth and FLOP counts predicted and empirically validated

Smaller models trained efficiently on fewer GPUs with comparable performance

Abstract

Energy efficiency of training and inferencing with large neural network models is a critical challenge facing the future of sustainable large-scale machine learning workloads. This paper introduces an alternative strategy, called phantom parallelism, to minimize the net energy consumption of traditional tensor (model) parallelism, the most energy-inefficient component of large neural network training. The approach is presented in the context of feed-forward network architectures as a preliminary, but comprehensive, proof-of-principle study of the proposed methodology. We derive new forward and backward propagation operators for phantom parallelism, implement them as custom autograd operations within an end-to-end phantom parallel training pipeline and compare its parallel performance and energy-efficiency against those of conventional tensor parallel training pipelines. Formal analyses that predict lower bandwidth and FLOP counts are presented with supporting empirical results on up to 256 GPUs that corroborate these gains. Experiments are shown to deliver approximately 50% reduction in the energy consumed to train FFNs using the proposed phantom parallel approach when compared with conventional tensor parallel methods. Additionally, the proposed approach is shown to train smaller phantom models to the same model loss on smaller GPU counts as larger tensor parallel models on larger GPU counts offering the possibility for even greater energy savings.