Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor

TL;DR

This paper introduces Cmprsr, a prompt compression method using smaller LLMs to reduce costs of large models, with a comprehensive benchmark and a new optimized model demonstrating superior performance and controllability across diverse inputs.

Contribution

It presents the first LLM-based prompt compressor benchmark, improves a vanilla compressor with meta-prompt optimization, and develops Cmprsr, a fine-tuned model with superior compression and generalization capabilities.

Findings

Cmprsr outperforms extractive and vanilla compression methods.

It closely follows the desired compression rate, enabling fine control.

Demonstrates strong performance across various datasets and input lengths.

Abstract

Motivated by the high costs of using black-box Large Language Models (LLMs), we introduce a novel prompt compression paradigm, under which we use smaller LLMs to compress inputs for the larger ones. We present the first comprehensive LLM-as-a-compressor benchmark spanning 25 open- and closed-source models, which reveals significant disparity in models' compression ability in terms of (i) preserving semantically important information (ii) following the user-provided compression rate (CR). We further improve the performance of gpt-4.1-mini, the best overall vanilla compressor, with Textgrad-based compression meta-prompt optimization. We also identify the most promising open-source vanilla LLM - Qwen3-4B - and post-train it with a combination of supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), pursuing the dual objective of CR adherence and maximizing the downstream task performance. We call the resulting model Cmprsr and demonstrate its superiority over both extractive and vanilla abstractive compression across the entire range of compression rates on lengthy inputs from MeetingBank and LongBench as well as short prompts from GSM8k. The latter highlights Cmprsr's generalizability across varying input lengths and domains. Moreover, Cmprsr closely follows the requested compression rate, offering fine control over the cost-quality trade-off.