Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference

TL;DR

This paper introduces EHPC, a prompt compression technique that uses evaluator heads in transformers to efficiently select key tokens, reducing computational costs and maintaining performance in long-context LLM tasks.

Contribution

The paper presents a novel, training-free prompt compression method leveraging evaluator heads to improve long-context inference efficiency in LLMs.

Findings

EHPC achieves state-of-the-art results on prompt compression benchmarks.

It significantly reduces inference costs and complexity.

EHPC performs competitively with key-value cache methods.

Abstract

Although applications involving long-context inputs are crucial for the effective utilization of large language models (LLMs), they also result in increased computational costs and reduced performance. To address this challenge, we propose an efficient, training-free prompt compression method that retains key information within compressed prompts. We identify specific attention heads in transformer-based LLMs, which we designate as evaluator heads, that are capable of selecting tokens in long inputs that are most significant for inference. Building on this discovery, we develop EHPC, an Evaluator Head-based Prompt Compression method, which enables LLMs to rapidly "skim through" input prompts by leveraging only the first few layers with evaluator heads during the pre-filling stage, subsequently passing only the important tokens to the model for inference. EHPC achieves state-of-the-art results across two mainstream benchmarks: prompt compression and long-context inference acceleration. Consequently, it effectively reduces the complexity and costs associated with commercial API calls. We further demonstrate that EHPC attains competitive results compared to key-value cache-based acceleration methods, thereby highlighting its potential to enhance the efficiency of LLMs for long-context tasks.