SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

TL;DR

SeedLM introduces a data-free, seed-based compression technique for LLMs that reduces memory access and speeds up inference, maintaining accuracy while significantly decreasing model size.

Contribution

The paper presents a novel post-training, seed-based compression method for LLMs that is data-free and generalizes across tasks, outperforming existing techniques in accuracy retention and inference speed.

Findings

SeedLM achieves better zero-shot accuracy at 4- and 3-bit quantization.

It maintains performance comparable to FP16 baselines.

FPGA tests show near 4x speed-up for 70B models.

Abstract

Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due to their high runtime cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of pseudo-random generators to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into a Linear Feedback Shift Register (LFSR) during inference to efficiently generate a random matrix. This matrix is then linearly combined with compressed coefficients to reconstruct the weight block. SeedLM reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory-bound tasks by trading compute for fewer memory accesses. Unlike state-of-the-art compression methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama 3 70B, which is particularly challenging to compress, show that SeedLM achieves significantly better zero-shot accuracy retention at 4- and 3-bit than state-of-the-art techniques, while maintaining performance comparable to FP16 baselines. Additionally, FPGA-based tests demonstrate that 4-bit SeedLM, as model size increases to 70B, approaches a 4x speed-up over an FP16 Llama 2/3 baseline.