CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization

TL;DR

CROP is a prompt optimization method that reduces token usage in large language models by regularizing response length, maintaining accuracy while significantly decreasing token consumption.

Contribution

It introduces a regularization technique in automatic prompt optimization to produce concise responses, improving token efficiency without sacrificing performance.

Findings

Achieved 80.6% reduction in token consumption on reasoning datasets.

Maintained competitive accuracy with only nominal performance decline.

Demonstrated effectiveness on GSM8K, LogiQA, and BIG-Bench Hard datasets.

Abstract

Large Language Models utilizing reasoning techniques improve task performance but incur significant latency and token costs due to verbose generation. Existing automatic prompt optimization(APO) frameworks target task accuracy exclusively at the expense of generating long reasoning traces. We propose Cost-Regularized Optimization of Prompts (CROP), an APO method that introduces regularization on response length by generating textual feedback in addition to standard accuracy feedback. This forces the optimization process to produce prompts that elicit concise responses containing only critical information and reasoning. We evaluate our approach on complex reasoning datasets, specifically GSM8K, LogiQA and BIG-Bench Hard. We achieved an 80.6\% reduction in token consumption while maintaining competitive accuracy, seeing only a nominal decline in performance. This presents a pragmatic solution for deploying token-efficient and cost-effective agentic AI systems in production pipelines.