LanguaShrink: Reducing Token Overhead with Psycholinguistics

TL;DR

LanguaShrink is a novel prompt compression framework inspired by psycholinguistics that significantly reduces prompt length and improves inference efficiency in large language models without sacrificing semantic integrity.

Contribution

It introduces a task-agnostic compression method using psycholinguistic principles, part-of-speech prioritization, and reinforcement learning, achieving up to 26x compression and faster inference.

Findings

Achieves up to 26 times prompt compression.

Improves end-to-end latency by 1.43 times.

Maintains semantic similarity across datasets.

Abstract

As large language models (LLMs) improve their capabilities in handling complex tasks, the issues of computational cost and efficiency due to long prompts are becoming increasingly prominent. To accelerate model inference and reduce costs, we propose an innovative prompt compression framework called LanguaShrink. Inspired by the observation that LLM performance depends on the density and position of key information in the input prompts, LanguaShrink leverages psycholinguistic principles and the Ebbinghaus memory curve to achieve task-agnostic prompt compression. This effectively reduces prompt length while preserving essential information. We referred to the training method of OpenChat.The framework introduces part-of-speech priority compression and data distillation techniques, using smaller models to learn compression targets and employing a KL-regularized reinforcement learning strategy for training.\cite{wang2023openchat} Additionally, we adopt a chunk-based compression algorithm to achieve adjustable compression rates. We evaluate our method on multiple datasets, including LongBench, ZeroScrolls, Arxiv Articles, and a newly constructed novel test set. Experimental results show that LanguaShrink maintains semantic similarity while achieving up to 26 times compression. Compared to existing prompt compression methods, LanguaShrink improves end-to-end latency by 1.43 times.