Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques

TL;DR

This survey reviews various model compression techniques like knowledge distillation, quantization, and pruning to enable efficient deployment of large language models on resource-limited devices.

Contribution

It provides a comprehensive overview of existing LLM compression methods, discussing their principles, variants, applications, and future directions.

Findings

Knowledge distillation effectively reduces model size.

Quantization maintains accuracy with lower precision.

Pruning simplifies models while preserving performance.

Abstract

Large Language Models (LLMs) have revolutionized many areas of artificial intelligence (AI), but their substantial resource requirements limit their deployment on mobile and edge devices. This survey paper provides a comprehensive overview of techniques for compressing LLMs to enable efficient inference in resource-constrained environments. We examine three primary approaches: Knowledge Distillation, Model Quantization, and Model Pruning. For each technique, we discuss the underlying principles, present different variants, and provide examples of successful applications. We also briefly discuss complementary techniques such as mixture-of-experts and early-exit strategies. Finally, we highlight promising future directions, aiming to provide a valuable resource for both researchers and practitioners seeking to optimize LLMs for edge deployment.