TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints

TL;DR

This paper introduces TRIAGE, a framework to evaluate whether language models can effectively plan and allocate resources under constraints, revealing significant gaps in their prospective metacognitive control abilities.

Contribution

The paper presents TRIAGE, a novel evaluation method for assessing language models' ability to plan and allocate resources under constraints, a previously unmeasured capability.

Findings

Current language models show substantial gaps in prospective metacognitive control.

Models' planning abilities vary across tasks like mathematics, science, and code generation.

The framework provides a quantitative measure of resource allocation efficiency.

Abstract

Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.