CarbonScaling: Extending Neural Scaling Laws for Carbon Footprint in Large Language Models

TL;DR

CarbonScaling introduces a comprehensive, hardware-aware analytical framework to accurately model and estimate the carbon footprint of large language model training, considering system-level factors and operational constraints.

Contribution

It extends neural scaling laws by integrating hardware heterogeneity, parallelism, and carbon accounting into a unified model for more precise emissions estimation.

Findings

Higher fidelity in carbon footprint estimation compared to regression baselines.

Embodied carbon becomes significant at trillion-parameter scales.

Framework supports modeling diverse parallelism strategies and hardware configurations.

Abstract

Large language models (LLMs) increasingly follow neural scaling laws that tie performance gains to rapidly expanding computational budgets, raising concerns about the sustainability of frontier-scale training. Existing carbon-estimation methods largely depend on regression over historical runs and fail to capture critical system-level factors, including hardware heterogeneity, distributed parallelism, communication overhead, and architectural sparsity. We present \textit{CarbonScaling}, a hardware-aware analytical framework for modeling the carbon scaling behavior of frontier LLM training. The framework integrates neural scaling laws, distributed training strategies, accelerator and interconnect modeling, and operational and embodied carbon accounting to estimate feasible hardware configurations and associated emissions. CarbonScaling jointly models tensor, pipeline, data, and expert parallelism while incorporating memory, bandwidth, utilization, and runtime constraints. Experimental validation demonstrates substantially higher fidelity than regression-based baselines and highlights the growing importance of embodied carbon at trillion-parameter scales. Source code: \url{https://github.com/UnchartedRLab/CarbonScaling}.