QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization

TL;DR

QwenLong-CPRS introduces a dynamic context optimization framework for large language models, significantly improving long-context processing efficiency and accuracy through novel compression and boundary-aware mechanisms.

Contribution

It presents a novel, natural language-guided dynamic context compression method with architecture-agnostic integration, achieving state-of-the-art performance on long-sequence benchmarks.

Findings

Achieves 21.59× context compression with 19.15-point performance gains.

Outperforms RAG and sparse attention methods in accuracy and efficiency.

Surpasses leading proprietary LLMs on long-context benchmarks.

Abstract

This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance. Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference. Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59$\times$ context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance.