Quad Length Codes for Lossless Compression of e4m3

TL;DR

This paper introduces Quad Length Codes, a hybrid lossless compression method for e4m3 data that balances compression efficiency with decoding speed, simplifying hardware implementation compared to Huffman codes.

Contribution

It proposes a novel hybrid coding scheme that uses a lookup table and area-based code lengths, improving decoding speed and hardware simplicity over traditional Huffman codes.

Findings

Achieves 13.9% compressibility for e4m3 data.

Significantly faster decoding than Huffman codes.

Simplifies hardware implementation.

Abstract

Training and serving Large Language Models (LLMs) relies heavily on parallelization and collective operations, which are frequently bottlenecked by network bandwidth. Lossless compression using e.g., Huffman codes can alleviate the issue, however, Huffman codes suffer from slow, bit-sequential decoding and high hardware complexity due to deep tree traversals. Universal codes e.g., Exponential-Golomb codes are faster to decode but do not exploit the symbol frequency distributions. To address these limitations, this paper introduces Quad Length Codes, a hybrid approach designed to balance compression efficiency with decoding speed. The coding scheme uses 3 prefix bits to divide the 256 symbols into 8 areas. Each area has a different code length and encodes a different number of symbols. The scheme uses a Look Up Table with 256 entries, significantly simplifying the hardware implementation compared to Huffman trees. The coding scheme can be adapted for different distributions. For the e4m3 data type, the scheme achieves a compressibility of 13.9% in comparison to 15.9% achieved by Huffman codes, but it significantly speeds up the decoding and simplifies the hardware complexity.