Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference

TL;DR

Endor introduces a hardware-friendly sparse format that compresses offloaded LLM weights, significantly reducing transfer latency and accelerating inference on resource-constrained platforms.

Contribution

The paper presents a novel sparse format for offloaded LLM inference that improves transfer efficiency and speedup, addressing GPU memory limitations.

Findings

Endor achieves up to 1.78x speedup on OPT-66B and Llama2-70B models.

Using Endor with SSD-to-GPU transfer yields up to 2.37x speedup.

The sparse format reduces transfer latency with high compression and low decompression overhead.

Abstract

The increasing size of large language models (LLMs) challenges their usage on resource-constrained platforms. For example, memory on modern GPUs is insufficient to hold LLMs that are hundreds of Gigabytes in size. Offloading is a popular method to escape this constraint by storing weights of an LLM model to host CPU memory and SSD, then loading each weight to GPU before every use. In our case study of offloaded inference, we found that due to the low bandwidth between storage devices and GPU, the latency of transferring large model weights from its offloaded location to GPU memory becomes the critical bottleneck with actual compute taking nearly 0% of runtime. To effectively reduce the weight transfer latency, we propose a novel sparse format that compresses the unstructured sparse pattern of pruned LLM weights to non-zero values with high compression ratio and low decompression overhead. Endor achieves this by expressing the positions of non-zero elements with a bitmap. Compared to offloaded inference using the popular Huggingface Accelerate, applying Endor accelerates OPT-66B by 1.70x and Llama2-70B by 1.78x. When direct weight transfer from SSD to GPU is leveraged, Endor achieves 2.25x speedup on OPT-66B and 2.37x speedup on Llama2-70B.