GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment

TL;DR

GMSA is a novel framework that compresses long context in NLP tasks by merging groups and aligning semantics across layers, reducing computational costs and redundancy while maintaining performance.

Contribution

GMSA introduces Group Merging and Layer Semantic Alignment to improve context compression and semantic fidelity in long-context NLP tasks.

Findings

GMSA outperforms existing soft prompt compression methods.

GMSA achieves better context reconstruction on benchmarks.

GMSA maintains low latency while improving performance.

Abstract

Large Language Models (LLMs) have achieved remarkable performance across a wide range of Natural Language Processing (NLP) tasks. However, in long-context scenarios, they face two challenges: high computational cost and information redundancy. To address these challenges, we propose GMSA, an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. GMSA introduces Group Merging to achieve more uniform aggregation, mitigating semantic dominance during autoencoder pretraining, and Layer Semantic Alignment (LSA) to bridge the semantic gap between high-level abstract semantics and low-level input semantics. We first pretrain GMSA as an autoencoder and then fine-tune it for downstream tasks. Experiments demonstrate that GMSA improves context reconstruction compared to existing soft prompt compression paradigm and outperforms baselines on multiple long-context question answering and summarization benchmarks across two backbone models, while maintaining low end-to-end latency.