Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability

TL;DR

Selection-p is a self-supervised prompt compression method that effectively reduces prompt size, maintains high task performance, and transfers well across models, addressing key limitations of prior approaches.

Contribution

It introduces a unified, self-supervised prompt compression technique that discretizes tokens and improves transferability without external data or model-specific tuning.

Findings

Achieves up to 10x compression with only 0.8% performance loss.

Outperforms prior methods in transferability across models.

Maintains performance on long-context in-context learning.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in a wide range of natural language processing tasks when leveraging in-context learning. To mitigate the additional computational and financial costs associated with in-context learning, several prompt compression methods have been proposed to compress the in-context learning prompts. Despite their success, these methods face challenges with transferability due to model-specific compression, or rely on external training data, such as GPT-4. In this paper, we investigate the ability of LLMs to develop a unified compression method that discretizes uninformative tokens, utilizing a self-supervised pre-training technique. By introducing a small number of parameters during the continual pre-training, the proposed Selection-p produces a probability for each input token, indicating whether to preserve or discard it. Experiments show Selection-p achieves state-of-the-art performance across numerous classification tasks, achieving compression rates of up to 10 times while experiencing only a marginal 0.8% decrease in performance. Moreover, it exhibits superior transferability to different models compared to prior work. Additionally, we further analyze how Selection-p helps maintain performance on in-context learning with long contexts.