Concept than Document: Context Compression via AMR-based Conceptual Entropy

TL;DR

This paper introduces an unsupervised method for compressing long contexts in language models using AMR graphs to retain essential semantics and reduce redundancy, improving accuracy and efficiency.

Contribution

It presents the first use of AMR-based conceptual entropy for context compression, enhancing semantic preservation in long document processing.

Findings

Outperforms baseline methods in accuracy on PopQA and EntityQuestions datasets.

Reduces context length significantly while maintaining or improving performance.

Demonstrates the effectiveness of AMR-based semantic filtering in LLM context management.

Abstract

Large Language Models (LLMs) face information overload when handling long contexts, particularly in Retrieval-Augmented Generation (RAG) where extensive supporting documents often introduce redundant content. This issue not only weakens reasoning accuracy but also increases computational overhead. We propose an unsupervised context compression framework that exploits Abstract Meaning Representation (AMR) graphs to preserve semantically essential information while filtering out irrelevant text. By quantifying node-level entropy within AMR graphs, our method estimates the conceptual importance of each node, enabling the retention of core semantics. Specifically, we construct AMR graphs from raw contexts, compute the conceptual entropy of each node, and screen significant informative nodes to form a condensed and semantically focused context than raw documents. Experiments on the PopQA and EntityQuestions datasets show that our method outperforms vanilla and other baselines, achieving higher accuracy while substantially reducing context length. To the best of our knowledge, this is the first work introducing AMR-based conceptual entropy for context compression, demonstrating the potential of stable linguistic features in context engineering.