Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation

TL;DR

PALACE is a user-side framework that accurately estimates hidden reasoning tokens in LLM API outputs, enabling reliable token auditing without internal access, thus promoting transparency and accountability.

Contribution

It introduces a novel reasoning token count estimation method using a lightweight domain router, addressing variance in token usage across diverse reasoning tasks.

Findings

Achieves low relative error in token estimation across multiple benchmarks.

Supports fine-grained cost auditing and inflation detection.

Demonstrates effectiveness in math, coding, medical, and general reasoning tasks.

Abstract

Commercial LLM services often conceal internal reasoning traces while still charging users for every generated token, including those from hidden intermediate steps, raising concerns of token inflation and potential overbilling. This gap underscores the urgent need for reliable token auditing, yet achieving it is far from straightforward: cryptographic verification (e.g., hash-based signature) offers little assurance when providers control the entire execution pipeline, while user-side prediction struggles with the inherent variance of reasoning LLMs, where token usage fluctuates across domains and prompt styles. To bridge this gap, we present PALACE (Predictive Auditing of LLM APIs via Reasoning Token Count Estimation), a user-side framework that estimates hidden reasoning token counts from prompt-answer pairs without access to internal traces. PALACE introduces a GRPO-augmented adaptation module with a lightweight domain router, enabling dynamic calibration across diverse reasoning tasks and mitigating variance in token usage patterns. Experiments on math, coding, medical, and general reasoning benchmarks show that PALACE achieves low relative error and strong prediction accuracy, supporting both fine-grained cost auditing and inflation detection. Taken together, PALACE represents an important first step toward standardized predictive auditing, offering a practical path to greater transparency, accountability, and user trust.