A Survey on Model Compression for Large Language Models

TL;DR

This survey reviews various model compression techniques like quantization, pruning, and distillation for large language models, emphasizing recent progress, benchmarking, and evaluation metrics to improve their efficiency and practical deployment.

Contribution

It provides a comprehensive overview of recent advancements in model compression for LLMs, including benchmarking strategies and evaluation metrics, to guide future research and application.

Findings

Summarizes recent advancements in compression techniques

Highlights benchmarking strategies for compressed LLMs

Discusses evaluation metrics for assessing compression effectiveness

Abstract

Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.