OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

TL;DR

OPTIMA introduces a scalable, one-shot pruning method for large language models that uses quadratic programming to optimize weight reconstruction, significantly improving accuracy without fine-tuning.

Contribution

The paper presents OPTIMA, a novel layer-wise quadratic programming approach for one-shot pruning that balances accuracy and scalability at large model scales.

Findings

Achieves up to 3.97% accuracy improvement over existing methods.

Prunes an 8B-parameter transformer in 40 hours on a single GPU.

Sets new state-of-the-art accuracy-efficiency trade-offs for one-shot pruning.

Abstract

Post-training model pruning is a promising solution, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principled joint optimization methods recover accuracy but are computationally infeasible at modern scale. One-shot methods such as SparseGPT offer a practical trade-off in optimality by applying efficient, approximate heuristic weight updates. To close this gap, we introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability. OPTIMA casts layer-wise weight reconstruction after mask selection as independent, row-wise Quadratic Programs (QPs) that share a common layer Hessian. Solving these QPs yields the per-row globally optimal update with respect to the reconstruction objective given the estimated Hessian. The shared-Hessian structure makes the problem highly amenable to batching on accelerators. We implement an accelerator-friendly QP solver that accumulates one Hessian per layer and solves many small QPs in parallel, enabling one-shot post-training pruning at scale on a single accelerator without fine-tuning. OPTIMA integrates with existing mask selectors and consistently improves zero-shot performance across multiple LLM families and sparsity regimes, yielding up to 3.97% absolute accuracy improvement. On an NVIDIA H100, OPTIMA prunes a 8B-parameter transformer end-to-end in 40 hours with 60GB peak memory. Together, these results set a new state-of-the-art accuracy-efficiency trade-offs for one-shot post-training pruning.