ML-EcoLyzer: Quantifying the Environmental Cost of Machine Learning Inference Across Frameworks and Hardware

TL;DR

ML-EcoLyzer is a comprehensive tool that measures and analyzes the environmental impact of machine learning inference across various frameworks and hardware, introducing the ESS metric to guide sustainable model deployment.

Contribution

It introduces ML-EcoLyzer, a novel cross-framework tool with an environmental sustainability score, providing extensive empirical data on inference costs across diverse models and hardware.

Findings

Quantization improves the Environmental Sustainability Score (ESS).

Large accelerators can be inefficient for lightweight models.

Small models can have high environmental costs if implemented poorly.

Abstract

Machine learning inference occurs at a massive scale, yet its environmental impact remains poorly quantified, especially on low-resource hardware. We present ML-EcoLyzer, a cross-framework tool for measuring the carbon, energy, thermal, and water costs of inference across CPUs, consumer GPUs, and datacenter accelerators. The tool supports both classical and modern models, applying adaptive monitoring and hardware-aware evaluation. We introduce the Environmental Sustainability Score (ESS), which quantifies the number of effective parameters served per gram of CO$_2$ emitted. Our evaluation covers over 1,900 inference configurations, spanning diverse model architectures, task modalities (text, vision, audio, tabular), hardware types, and precision levels. These rigorous and reliable measurements demonstrate that quantization enhances ESS, huge accelerators can be inefficient for lightweight applications, and even small models may incur significant costs when implemented suboptimally. ML-EcoLyzer sets a standard for sustainability-conscious model selection and offers an extensive empirical evaluation of environmental costs during inference.