EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices

TL;DR

EntroLLM introduces a novel entropy-based weight compression method combining mixed quantization and Huffman coding, enabling efficient large language model inference on edge devices with significant storage and speed improvements.

Contribution

The paper presents EntroLLM, a compression framework that enhances weight compressibility and inference efficiency without retraining, suitable for deployment on resource-constrained edge hardware.

Findings

Tensor-level quantization increases weight compressibility and improves Huffman encoding by up to 11.3x.

Up to 30% storage savings over uint8 and 65% over uint4 models.

Achieves 31.9-146.6% faster inference on edge devices like NVIDIA JETSON P3450.

Abstract

Large Language Models (LLMs) achieve strong performance across tasks, but face storage and compute challenges on edge devices. We propose EntroLLM, a compression framework combining mixed quantization and entropy coding to reduce storage while preserving accuracy. We use a combination of unsigned and asymmetric quantization. Tensor-level quantization produces an entropy-reducing effect, increasing weight compressibility, and improving downstream Huffman encoding by $7\times$ (8-bit) and $11.3\times$ (4-bit) over state-of-the-art methods. Huffman coding further reduces memory bandwidth demands, while a parallel decoding strategy enables efficient weight retrieval with minimal latency. Experiments on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) show up to $30\%$ storage savings over uint8 and $65\%$ over uint4 models, with $31.9-146.6\%$ faster inference on memory-limited devices like the NVIDIA JETSON P3450. EntroLLM requires no retraining and is compatible with existing post-training quantization pipelines, making it practical for edge LLM deployment.