How Green are Neural Language Models? Analyzing Energy Consumption in Text Summarization Fine-tuning

TL;DR

This paper evaluates the environmental impact of fine-tuning neural language models for text summarization, highlighting the significant carbon footprint of larger models and advocating for energy-efficient AI practices.

Contribution

It provides a comparative analysis of energy consumption and performance trade-offs among three neural language models during fine-tuning for summarization tasks.

Findings

LLaMA-3-8B has the highest carbon footprint among the models.

Performance metrics vary across models, with larger models generally performing better.

Environmental impact should be considered in neural language model development.

Abstract

Artificial intelligence systems significantly impact the environment, particularly in natural language processing (NLP) tasks. These tasks often require extensive computational resources to train deep neural networks, including large-scale language models containing billions of parameters. This study analyzes the trade-offs between energy consumption and performance across three neural language models: two pre-trained models (T5-base and BART-base), and one large language model (LLaMA-3-8B). These models were fine-tuned for the text summarization task, focusing on generating research paper highlights that encapsulate the core themes of each paper. The carbon footprint associated with fine-tuning each model was measured, offering a comprehensive assessment of their environmental impact. It is observed that LLaMA-3-8B produces the largest carbon footprint among the three models. A wide range of evaluation metrics, including ROUGE, METEOR, MoverScore, BERTScore, and SciBERTScore, were employed to assess the performance of the models on the given task. This research underscores the importance of incorporating environmental considerations into the design and implementation of neural language models and calls for the advancement of energy-efficient AI methodologies.