Semantic Retention and Extreme Compression in LLMs: Can We Have Both?

TL;DR

This paper explores combining pruning and quantization for LLM compression, introducing a new metric to balance semantic retention and compression, and demonstrates improved performance over single-method approaches.

Contribution

It presents a novel joint compression framework for LLMs, including a new metric SrCr to evaluate semantic retention, and shows superior performance-to-compression ratios.

Findings

20% performance increase over quantization-only models at same compression rate

Introduction of SrCr metric for better evaluation of semantic retention

Joint pruning and quantization outperform single-method compression approaches

Abstract

The exponential growth in Large Language Model (LLM) deployment has intensified the need for efficient model compression techniques to reduce computational and memory costs. While pruning and quantization have shown promise, their combined potential remains largely unexplored. In this paper, we examine joint compression and how strategically combining pruning and quantization could yield superior performance-to-compression ratios compared to single-method approaches. Recognizing the challenges in accurately assessing LLM performance, we address key limitations of previous evaluation frameworks and introduce the Semantic Retention Compression Rate (SrCr), a novel metric that quantifies the trade-off between model compression and semantic preservation, facilitating the optimization of pruning-quantization configurations. Experiments demonstrate that our recommended combination achieves, on average, a 20% performance increase compared to an equivalent quantization-only model at the same theoretical compression rate.