CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization

TL;DR

This paper presents heirname, a pioneering framework for co-optimizing Transformer models and hardware accelerators with a focus on reducing carbon footprint, balancing sustainability with performance.

Contribution

It introduces the first framework integrating operational and embodied carbon considerations into Transformer and hardware co-design, enabling more sustainable AI development.

Findings

Potential to reduce total carbon emissions by up to 30%

Maintains accuracy and latency while optimizing for carbon efficiency

Demonstrates effectiveness across various Transformer models

Abstract

Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to increase the associated lifecycle carbon footprint, including both \emph{operational carbon} from training and inference and \emph{embodied carbon} from AI hardware manufacturing. We introduce \ourframework -- the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, \ourframework enables sustainability-driven model architecture and hardware accelerator co-design that reveals fundamentally different trade-offs than latency- or energy-centric approaches. Evaluated across a range of Transformer models, \ourframework consistently demonstrates the potential to reduce total carbon emissions -- by up to 30\% -- while maintaining accuracy and latency. We further highlight its extensibility through a focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of \ourframework is available at {\small{\href{https://github.com/facebookresearch/CATransformers}{\texttt{https://github.com/facebookresearch/CATransformers}}}}.