The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts

TL;DR

This paper investigates the perplexity paradox in large language models, demonstrating how code and math prompts behave differently under compression, and introduces an adaptive compression method that reduces costs while maintaining quality.

Contribution

It validates the perplexity paradox across multiple benchmarks, analyzes token-level perplexity differences, and proposes the TAAC adaptive compression algorithm.

Findings

Code syntax tokens have high perplexity and are preserved during compression.

Numerical values in math problems have low perplexity and are often pruned.

TAAC achieves significant cost reduction with high quality preservation.

Abstract

In "Compress or Route?" (Johnson, 2026), we found that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually. That study was limited to HumanEval (164 problems), left the "perplexity paradox" mechanism unvalidated, and provided no adaptive algorithm. This paper addresses all three gaps. First, we validate across six code benchmarks (HumanEval, MBPP, HumanEval+, MultiPL-E) and four reasoning benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-STEM), confirming the compression threshold generalizes across languages and difficulties. Second, we conduct the first per-token perplexity analysis (n=723 tokens), revealing a "perplexity paradox": code syntax tokens are preserved (high perplexity) while numerical values in math problems are pruned despite being task-critical (low perplexity). Signature injection recovers +34 percentage points in pass rate (5.3% to 39.3%; Cohen's h=0.890). Third, we propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%. MBPP validation (n=1,800 trials) confirms systematic variation: 3.6% at r=0.3 to 54.6% at r=1.0.