Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression

TL;DR

Sentinel introduces a novel context compression method for LLMs that decodes which parts of the context are utilized during inference, enabling efficient retrieval-augmented generation with significant compression gains.

Contribution

Sentinel leverages attention probing to decode context relevance directly from a frozen LLM, enabling effective compression without supervised relevance metrics.

Findings

Achieves up to 5x compression on LongBench with maintained QA performance.

Generalizes well across languages and out-of-domain data.

Uses a lightweight proxy model for decoding relevance signals.

Abstract

Retrieval-augmented generation (RAG) often suffers from long and noisy retrieved contexts. Prior context compression methods rely on predefined importance metrics or supervised compression models, rather than on the model's own inference-time behavior. We propose Sentinel, a lightweight sentence-level compression framework that treats context compression as an understanding decoding problem. Sentinel probes native attention behaviors of a frozen LLM with a lightweight readout to decode which parts of the context are actually utilized when answering a query, rather than using attention as a direct relevance score. We empirically observe that decoded relevance signals exhibit sufficient consistency across model scales to support effective compression with compact proxy models. On LongBench, Sentinel with a 0.5B proxy model achieves up to 5x compression while matching the QA performance of 7B-scale baselines, and despite being trained only on English QA data, generalizes effectively to Chinese and out-of-domain settings.