Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor

TL;DR

This paper introduces a task-agnostic prompt compression framework that generates concise, relevant prompts without task-specific templates, leveraging reinforcement learning and context-aware sentence embeddings, outperforming existing methods on benchmarks.

Contribution

We propose a novel, task-agnostic prompt compression method that does not rely on handcrafted templates, using reinforcement learning to optimize relevance across diverse tasks and domains.

Findings

The largest model outperforms state-of-the-art on LongBench and ZeroSCROLLS.

The smallest model achieves comparable performance with significantly fewer parameters.

Our approach generalizes prompt compression across tasks without explicit question templates.

Abstract

The rise of Large Language Models (LLMs) has led to significant interest in prompt compression, a technique aimed at reducing the length of input prompts while preserving critical information. However, the prominent approaches in prompt compression often require explicit questions or handcrafted templates for compression, limiting their generalizability. We propose Task-agnostic Prompt Compression (TPC), a novel framework that generalizes compression across tasks and domains without requiring input questions or templates. TPC generates a context-relevant task description using a task descriptor trained on a curated dataset of context and query pairs, and fine-tuned via reinforcement learning with a reward function designed to capture the most relevant information. The task descriptor is then utilized to compute the relevance of each sentence in the prompt to generate the compressed prompt. We introduce 3 model sizes (Base, Large, and Huge), where the largest model outperforms the existing state-of-the-art methods on LongBench and ZeroSCROLLS benchmarks, and our smallest model performs comparable to the existing solutions while being considerably smaller.