Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agents

TL;DR

This paper introduces Entropic Context Shaping (ECS), an information-theoretic method for selecting useful context for LLMs that outperforms lexical similarity approaches by focusing on pragmatic utility.

Contribution

ECS provides a novel framework that measures context utility through distribution shifts, enabling more effective context selection for LLMs.

Findings

ECS with Llama-3.1-8B achieves 71.83% relative improvement over TF-IDF.

ECS effectively captures pragmatic utility over lexical similarity.

Theoretical analysis shows near-zero shift for irrelevant updates.

Abstract

Context engineering for large language model (LLM) agents requires distinguishing pragmatically useful information from misleading distractors. We introduce Entropic Context Shaping (ECS), an information-theoretic framework that measures context utility via the shift in the model's answer distribution toward the correct answer. Unlike lexical similarity methods that rely on word overlap, ECS captures pragmatic utility -- whether a passage actually helps answer the question. We formalize utility as the signed change in answer probability and provide theoretical analysis showing that task-irrelevant updates yield near-zero distribution shift. We evaluate on multi-turn context selection tasks using LongMemEval (session-level) and LoCoMo (turn-level) benchmarks. On fine-grained turn selection, ECS with Llama-3.1-8B achieves F1=0.265, a 71.83% relative improvement over TF-IDF (F1=0.154), demonstrating that pragmatic utility outperforms lexical similarity when precise context selection matters. Code and data are available in the supplementary materials.