KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

TL;DR

This paper introduces KV-Latent, a method to reduce KV cache size in Transformer-based LLMs by down-sampling vectors into a latent space, enhancing inference efficiency with minimal additional training.

Contribution

The paper proposes a novel KV-Latent paradigm that significantly reduces KV cache size and improves inference speed with less than 1% extra training, and enhances Rotary Positional Embedding stability.

Findings

Significant reduction in KV cache footprint.

Improved inference speed with minimal training overhead.

Enhanced stability of Rotary Positional Embedding.

Abstract

Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI. Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value (KV) cache during inference has emerged as a primary efficiency bottleneck, both in aspects of memory consumption and data transfer bandwidth limitations. To address these challenges, we propose a paradigm called KV-Latent. By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed, only with a small amount of extra training, less than 1\% of pre-training takes. Besides, we enhanced the stability of Rotary Positional Embedding applied on lower-dimensional vectors by modifying its frequency sampling mechanism, avoiding noise introduced by higher frequencies while retaining position attenuation. Our experiments, including both models with Grouped Query Attention and those without, have yielded satisfactory results. Finally, we conducted comparative experiments to study the impact of separately reducing Key and Value components on model's performance. Our approach allows for the construction of more efficient language model systems, and opens the new possibility on KV Cache saving and efficient LLMs. Our code is available at https://github.com/ShiLuohe/KV-Latent.