Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

TL;DR

This survey comprehensively reviews techniques to improve resource efficiency in large language models across their lifecycle, addressing computational, memory, energy, and financial challenges to promote sustainable AI development.

Contribution

It introduces a detailed categorization of resource efficiency methods, evaluation metrics, and datasets, providing a foundational reference for future research in sustainable LLMs.

Findings

Categorized resource efficiency techniques by resource type and lifecycle stage

Presented standardized evaluation metrics and datasets for fair comparison

Identified open research challenges and future directions

Abstract

The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.