Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression

TL;DR

Ecco is an entropy-aware cache compression method for LLMs that significantly reduces memory and latency overheads while maintaining accuracy, enabling more efficient deployment of large-scale models.

Contribution

Ecco introduces a novel entropy-based cache compression technique combining group-wise quantization, shared patterns, and parallel Huffman decoding for LLMs.

Findings

Achieves up to 2.9× speedup over state-of-the-art methods.

Increases memory capacity by nearly 4× without accuracy loss.

Reduces Huffman decoding latency by two orders of magnitude.

Abstract

Large language models (LLMs) have demonstrated transformative capabilities across diverse artificial intelligence applications, yet their deployment is hindered by substantial memory and computational demands, especially in resource-constrained environments. Quantization techniques have emerged as a critical solution, reducing data precision to enhance memory and computational efficiency. However, existing methods often suffer from high runtime overheads and potential accuracy degradation. To address these challenges, we propose Ecco, an entropy-based cache compression technique tailored for LLMs. Ecco combines group-wise and non-uniform quantization with pre-defined shared k-means patterns and Huffman coding to exploit the inherent entropy characteristics of LLM cache data. Recognizing the inefficiencies of traditional Huffman coding in terms of parallelism and latency, we introduce a novel parallel Huffman-based decoding process with a multi-stage pipeline design, reducing latency by two orders of magnitude and achieving throughput comparable to GPU L2 caches. Comprehensive evaluations demonstrate that Ecco achieves an up to 2.9$\times$ and 1.9$\times$ speedup over the state-of-the-art AWQ and SmoothQuant framework, 2.4$\times$ over the Olive accelerator, all while increasing memory capacity by nearly 4$\times$ and maintaining state-of-the-art LLM accuracy. These results underscore the effectiveness of our entropy-based cache compression in enhancing LLM performance and efficiency, paving the way for more deployable large-scale AI models.