Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs

TL;DR

This paper introduces a method to replace lengthy system prompts in large language models with a single token, significantly reducing inference costs while maintaining nearly the same task performance.

Contribution

A novel three-stage training framework that learns a single Behavior-Equivalent token to replace long prompts without model internals or labeled data.

Findings

Up to 3000x reduction in prompt length

Retains about 98% of original performance

Reduces inference cost significantly

Abstract

Carefully engineered system prompts play a critical role in guiding the behavior of LLM agents, but their considerable length introduces significant drawbacks, including increased inference latency, higher computational cost, and reduced effective context length. This raises the question of whether such lengthy prompts can be replaced by a drastically reduced number of tokens while preserving their behavioral effect on downstream tasks. To enable this, we propose a lightweight three-stage training framework that learns a single prompt-specific Behavior-Equivalent token ([BE]). The framework first trains [BE] to encode the natural-language content of the original system prompt via reconstruction, and then distills the prompt 's downstream behavior into this single token. Importantly, our method requires no access to model internals, no auxiliary compression models, and no labeled responses. Empirical evaluations on three datasets show that a single [BE] token achieves up to a 3000x reduction in prompt length, while retaining about 98% of the downstream performance of the original system prompts. This substantially reduces inference cost and leaves almost the entire context window available for user inputs.