Progressive Binarization with Semi-Structured Pruning for LLMs

TL;DR

This paper introduces PBS$^2$P, a novel framework combining progressive binarization and semi-structured pruning to effectively compress large language models while maintaining high performance.

Contribution

The paper proposes a new post-training compression method that jointly optimizes binarization and pruning, improving stability and accuracy over existing techniques.

Findings

Outperforms state-of-the-art binary quantization methods in perplexity.

Achieves higher downstream task accuracy.

Demonstrates effectiveness across multiple LLM families.

Abstract

Large language models (LLMs) have achieved remarkable progress in natural language processing, but their high computational and memory costs hinder deployment on resource-constrained devices. Binarization represents the most extreme form of quantization, yet binarized models still contain redundancy that can be further removed. Pruning provides a natural way to eliminate such redundancy, but na\"ive combination with binarization often results in severe performance degradation. In this paper, we propose Progressive Binarization with Semi-Structured Pruning (PBS$^2$P), a novel post-training framework that seamlessly integrates binarization and semi-structured pruning. We first propose Stepwise semi-structured Pruning with Binarization Optimization (SPBO), which progressively introduces sparsity while optimizing binarization parameters to jointly reduce pruning and quantization error, yielding more stable and accurate compression. Additionally, we propose a Coarse-to-Fine Search (CFS) that first allocates pruning ratios and then refines element selection, further enhancing overall performance. Extensive experiments across multiple LLM families show that PBS$^2$P consistently outperforms state-of-the-art (SOTA) binary post-training quantization methods in both perplexity and downstream accuracy. The code and models will be available at https://github.com/XIANGLONGYAN/PBS2P.