Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model

TL;DR

This paper quantifies the carbon footprint of training and deploying BLOOM, a 176-billion parameter language model, highlighting environmental impacts and challenges in precise measurement.

Contribution

It provides the first comprehensive estimate of BLOOM's carbon emissions across its entire life cycle, including training and inference.

Findings

Training emitted approximately 24.7 tonnes of CO2 considering only dynamic power.

Total lifecycle emissions, including manufacturing, are about 50.5 tonnes of CO2.

Deployment for inference also contributes significantly to overall emissions.

Abstract

Progress in machine learning (ML) comes with a cost to the environment, given that training ML models requires significant computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of~\carboneq~if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also study the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting.