HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

TL;DR

HashEvict is a novel LSH-based algorithm that compresses the KV cache in transformer models, reducing memory usage by 30-70% with minimal performance loss across various NLP tasks.

Contribution

It introduces a pre-attention, dynamic cache eviction strategy using locality-sensitive hashing to efficiently compress KV caches in large language models.

Findings

Compresses KV cache by 30%-70%.

Maintains high performance in multiple NLP tasks.

Reduces computational costs compared to attention-based methods.

Abstract

Transformer-based large language models (LLMs) use the key-value (KV) cache to significantly accelerate inference by storing the key and value embeddings of past tokens. However, this cache consumes significant GPU memory. In this work, we introduce HashEvict, an algorithm that uses locality-sensitive hashing (LSH) to compress the KV cache. HashEvict quickly locates tokens in the cache that are cosine dissimilar to the current query token. This is achieved by computing the Hamming distance between binarized Gaussian projections of the current token query and cached token keys, with a projection length much smaller than the embedding dimension. We maintain a lightweight binary structure in GPU memory to facilitate these calculations. Unlike existing compression strategies that compute attention to determine token retention, HashEvict makes these decisions pre-attention, thereby reducing computational costs. Additionally, HashEvict is dynamic - at every decoding step, the key and value of the current token replace the embeddings of a token expected to produce the lowest attention score. We demonstrate that HashEvict can compress the KV cache by 30%-70% while maintaining high performance across reasoning, multiple-choice, long-context retrieval and summarization tasks.