Aggressive Post-Training Compression on Extremely Large Language Models

TL;DR

This paper introduces a novel post-training compression method for large language models that achieves high sparsity and low-bit quantization, significantly reducing size with minimal accuracy loss, enabling deployment on personal devices.

Contribution

The paper presents a new network pruning and quantization technique that compresses large language models efficiently while preserving accuracy, facilitating practical deployment on resource-constrained devices.

Findings

Achieves over 0.7 sparsity and less than 8-bit quantization.

Compresses large language models within a few hours.

Maintains relatively small accuracy loss during compression.

Abstract

The increasing size and complexity of Large Language Models (LLMs) pose challenges for their deployment on personal computers and mobile devices. Aggressive post-training model compression is necessary to reduce the models' size, but it often results in significant accuracy loss. To address this challenge, we propose a novel network pruning technology that utilizes over 0.7 sparsity and less than 8 bits of quantization. Our approach enables the compression of prevailing LLMs within a couple of hours while maintaining a relatively small accuracy loss. In experimental evaluations, our method demonstrates effectiveness and potential for practical deployment. By making LLMs available on domestic devices, our work can facilitate a new era of natural language processing applications with wide-ranging impacts.