Environmental Footprint of GenAI Research: Insights from the Moshi Foundation Model

TL;DR

This paper analyzes the environmental impact of developing the Moshi 7B-parameter MLLM, providing detailed insights into compute usage and offering guidelines for more sustainable AI research.

Contribution

It offers the first fine-grained analysis of compute and environmental impacts throughout the entire research process of a large language model.

Findings

Quantifies GPU-time spent on different model components and phases.

Assesses energy, water, and resource depletion from hardware production and use.

Provides actionable guidelines to reduce environmental impacts of MLLM research.

Abstract

New multi-modal large language models (MLLMs) are continuously being trained and deployed, following rapid development cycles. This generative AI frenzy is driving steady increases in energy consumption, greenhouse gas emissions, and a plethora of other environmental impacts linked to datacenter construction and hardware manufacturing. Mitigating the environmental consequences of GenAI remains challenging due to an overall lack of transparency by the main actors in the field. Even when the environmental impacts of specific models are mentioned, they are typically restricted to the carbon footprint of the final training run, omitting the research and development stages. In this work, we explore the impact of GenAI research through a fine-grained analysis of the compute spent to create Moshi, a 7B-parameter speech-text foundation model for real-time dialogue developed by Kyutai, a leading privately funded open science AI lab. For the first time, our study dives into the anatomy of compute-intensive MLLM research, quantifying the GPU-time invested in specific model components and training phases, as well as early experimental stages, failed training runs, debugging, and ablation studies. Additionally, we assess the environmental impacts of creating Moshi from beginning to end using a life cycle assessment methodology: we quantify energy and water consumption, greenhouse gas emissions, and mineral resource depletion associated with the production and use of datacenter hardware. Our detailed analysis allows us to provide actionable guidelines to reduce compute usage and environmental impacts of MLLM research, paving the way for more sustainable AI research.