LEXI: Lossless Exponent Coding for Efficient Inter-Chiplet Communication in Hybrid LLMs

TL;DR

LEXI is a lossless exponent compression scheme that significantly reduces inter-chiplet communication overheads in hybrid LLMs, leading to notable latency improvements without impacting accuracy.

Contribution

The paper introduces LEXI, a novel Huffman-based lossless exponent compression method tailored for efficient inter-chiplet communication in large language models.

Findings

Reduces inter-chiplet communication by 33-45%.

Decreases end-to-end inference latency by 30-35%.

Imposes only 0.09% area and energy overheads.

Abstract

Data movement overheads increase the inference latency of state-of-the-art large language models (LLMs). These models commonly use the bfloat16 (BF16) format for stable training. Floating-point standards allocate eight bits to the exponent, but our profiling reveals that exponent streams exhibit fewer than 3 bits Shannon entropy, indicating high inherent compressibility. To exploit this potential, we propose LEXI, a novel lossless exponent compression scheme based on Huffman coding. LEXI compresses activations and caches on the fly while storing compressed weights for just-in-time decompression near compute, without sacrificing system throughput and model accuracy. The codecs at the ingress and egress ports of network-on-chip routers sustain the maximum link bandwidth via multi-lane LUT decoders, incurring only 0.09 percent area and energy overheads with GF 22 nm technology. LEXI reduces inter-chiplet communication and end-to-end inference latencies by 33-45 percent and 30-35 percent on modern Jamba, Zamba, and Qwen LLMs implemented on a homogeneous chiplet architecture.