UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

TL;DR

UniComp provides a comprehensive framework for evaluating various LLM compression methods across multiple dimensions, revealing insights into their effects on knowledge retention, reliability, and task-specific performance.

Contribution

It introduces a unified evaluation framework for pruning, quantization, and distillation, including diverse benchmarks and hardware-aware efficiency analysis.

Findings

Factual recall is largely preserved after compression.

Multi-step reasoning and multilingual capabilities degrade.

Task-specific calibration improves reasoning performance by up to 50%.

Abstract

Model compression is increasingly essential for deploying large language models (LLMs), yet existing comparative studies largely focus on pruning and quantization evaluated primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through evaluation of six compression techniques across 40 datasets, we observe (i) a consistent knowledge bias, where factual recall is largely preserved while multi-step reasoning, multilingual, and instruction-following capabilities degrade; (ii) a decoupling between performance and reliability, indicating that retained performance does not consistently imply preserved reliability; and (iii) that task-specific calibration can yield up to 50% relative improvement of reasoning performance in pruned models.