Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression

TL;DR

This paper investigates how the sequence of applying pruning and quantization affects joint model compression, providing theoretical insights and empirical validation to optimize compression order for better model performance.

Contribution

It introduces the Progressive Intensity Hypothesis and offers theoretical guarantees, advancing understanding of compression order effects in joint model compression.

Findings

Compression order significantly impacts model performance.

Weaker perturbations should be applied before stronger ones.

The hypothesis holds across language and vision models, multi-stage, and mixed-precision setups.

Abstract

What happens when multiple compression methods are combined-does the order in which they are applied matter? Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization. A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline. Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases. Consequently, the broader role of compression order in shaping model performance remains poorly understood. In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis. We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones. We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap. Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.