A Systematic Study of Compression Ordering for Large Language Models

TL;DR

This paper systematically investigates how the order of applying compression techniques like pruning, distillation, and quantization affects the performance and compression ratio of large language models, providing practical guidelines for efficient deployment.

Contribution

It introduces a comprehensive analysis of compression technique sequences for LLMs, highlighting the optimal order for balancing compression and performance.

Findings

Quantization achieves the highest standalone compression.

Pruning causes moderate quality degradation.

Pruning, distillation, then quantization (P-KD-Q) yields the best balance.

Abstract

Large Language Models (LLMs) require substantial computational resources, making model compression essential for efficient deployment in constrained environments. Among the dominant compression techniques: knowledge distillation, structured pruning, and low-bit quantization, their individual effects are well studied, but their interactions and optimal sequencing remain unclear. This work systematically examines how these techniques perform both independently and in combination when applied to the Qwen2.5 3B model. We evaluate multiple compression pipelines, including single, and proposed three-technique sequences, using perplexity, G-Eval, clarity, prompt alignment, and compression ratio as metrics. Our experiments show that quantization provides the greatest standalone compression, while pruning introduces moderate quality degradation. Critically, the ordering of techniques significantly affects the final model quality: the sequence Pruning, Knowledge Distillation, Quantization (P-KD-Q) yields the best balance, achieving a 3.68x compression ratio while preserving strong instruction-following and language understanding capabilities. Conversely, pipelines applying quantization early suffer severe performance degradation due to irreversible information loss that impairs subsequent training. Overall, this study offers practical insight into designing effective, ordering-aware compression pipelines for deploying LLMs in resource-limited settings.