Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference

TL;DR

Huff-LLM introduces an end-to-end lossless compression method for large language models, enabling efficient storage, reduced bandwidth, and improved inference latency and energy efficiency on edge devices.

Contribution

It presents a novel lossless compression technique for LLMs that preserves model behavior and enhances deployment efficiency across various hardware platforms.

Findings

Enables storage of larger models in memory

Reduces bandwidth for weight loading

Improves inference latency and energy efficiency

Abstract

As they become more capable, large language models (LLMs) have continued to rapidly increase in size. This has exacerbated the difficulty in running state of the art LLMs on small, edge devices. Standard techniques advocate solving this problem through lossy compression techniques such as quantization or pruning. However, such compression techniques are lossy, and have been shown to change model behavior in unpredictable manners. We propose Huff-LLM, an \emph{end-to-end, lossless} model compression method that lets users store LLM weights in compressed format \emph{everywhere} -- cloud, disk, main memory, and even in on-chip memory/buffers. This allows us to not only load larger models in main memory, but also reduces bandwidth required to load weights on chip, and makes more efficient use of on-chip weight buffers. In addition to the memory savings achieved via compression, we also show latency and energy efficiency improvements when performing inference with the compressed model.