AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees

TL;DR

AdmTree introduces an adaptive hierarchical framework that compresses long text contexts efficiently while preserving semantic richness, addressing limitations of existing methods in LLMs processing long sequences.

Contribution

The paper presents AdmTree, a novel adaptive semantic tree approach that dynamically segments and summarizes long contexts with minimal additional parameters, enhancing semantic preservation and efficiency.

Findings

Maintains high semantic fidelity in long context processing.

Reduces computational complexity of self-attention in LLMs.

Effectively balances detail preservation with global coherence.

Abstract

The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts, a capability essential for many advanced applications. Context compression aims to alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, or an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with a central focus on preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based on information density, utilizing gist tokens to summarize variable-length segments as the leaves of a semantic binary tree. This structure, together with a lightweight aggregation mechanism and a frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.