Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters
Zhiyu Guo, Hidetaka Kamigaito, Taro Watanabe

TL;DR
This paper introduces Value-Aware Token Pruning (VATP), a method that improves large language model efficiency by combining attention scores and value vector norms for token importance, leading to better cache reduction.
Contribution
The paper proposes a novel token pruning method that incorporates value vector norms alongside attention scores, addressing limitations of previous approaches that relied solely on attention scores.
Findings
VATP outperforms attention-score-only methods in over 12 of 16 tasks.
Incorporating value vector norms improves token importance evaluation.
Extensive experiments validate the effectiveness of VATP across multiple benchmarks.
Abstract
Scaling the context size of large language models (LLMs) enables them to perform various new tasks, e.g., book summarization. However, the memory cost of the Key and Value (KV) cache in attention significantly limits the practical applications of LLMs. Recent works have explored token pruning for KV cache reduction in LLMs, relying solely on attention scores as a token importance indicator. However, our investigation into value vector norms revealed a notably non-uniform pattern questioning their reliance only on attention scores. Inspired by this, we propose a new method: Value-Aware Token Pruning (VATP) which uses both attention scores and the norm of value vectors to evaluate token importance. Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B across 16 LongBench tasks demonstrate that VATP outperforms attention-score-only baselines in over 12 tasks, confirming…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsCaching and Content Delivery · Advanced Wireless Network Optimization · Distributed and Parallel Computing Systems
MethodsSoftmax · Attention Is All You Need · Pruning
