Quad Length Codes for Lossless Compression of e4m3
Aditya Agrawal, Albert Magyar, Hiteshwar Eswaraiah, Patrick Sheridan, Pradeep Janedula, Ravi Krishnan Venkatesan, Krishna Nair, Ravi Iyer

TL;DR
This paper introduces Quad Length Codes, a hybrid lossless compression method for e4m3 data that balances compression efficiency with decoding speed, simplifying hardware implementation compared to Huffman codes.
Contribution
It proposes a novel hybrid coding scheme that uses a lookup table and area-based code lengths, improving decoding speed and hardware simplicity over traditional Huffman codes.
Findings
Achieves 13.9% compressibility for e4m3 data.
Significantly faster decoding than Huffman codes.
Simplifies hardware implementation.
Abstract
Training and serving Large Language Models (LLMs) relies heavily on parallelization and collective operations, which are frequently bottlenecked by network bandwidth. Lossless compression using e.g., Huffman codes can alleviate the issue, however, Huffman codes suffer from slow, bit-sequential decoding and high hardware complexity due to deep tree traversals. Universal codes e.g., Exponential-Golomb codes are faster to decode but do not exploit the symbol frequency distributions. To address these limitations, this paper introduces Quad Length Codes, a hybrid approach designed to balance compression efficiency with decoding speed. The coding scheme uses 3 prefix bits to divide the 256 symbols into 8 areas. Each area has a different code length and encodes a different number of symbols. The scheme uses a Look Up Table with 256 entries, significantly simplifying the hardware implementation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAlgorithms and Data Compression · Error Correcting Code Techniques · Advanced Data Compression Techniques
