LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling

TL;DR

This paper introduces a metacognitive harness inspired by cognitive psychology to enable LLMs to better control their reasoning process during inference, significantly improving accuracy without additional training.

Contribution

It proposes a novel control framework that leverages LLMs' self-monitoring signals to decide when to trust, retry, or finalize answers, enhancing reasoning performance.

Findings

Improved accuracy from 48.3% to 56.9% on benchmarks.

Outperforms strongest leaderboard entries on multiple evaluation settings.

Harness works without parameter updates or fine-tuning.

Abstract

Large language models (LLMs) often expose useful signals of self-monitoring: before solving a problem, they can estimate whether they are likely to succeed, and after solving it, they can judge whether their answer is likely to be correct. However, these signals are typically measured or elicited in isolation, rather than used to control inference. In this work, we ask whether LLMs possess latent metacognitive ability that can be turned into effective test-time control. Inspired by the Nelson--Narens theory from cognitive psychology, we propose a metacognitive harness that separates monitoring from reasoning. For each problem, the model first reports a pre-solve feeling-of-knowing (FOK) signal; after each solve attempt, it reports a post-solve judgment-of-learning (JOL) signal. Rather than treating these signals as passive confidence estimates, the harness turns them into an explicit control interface for reasoning: it decides when to trust the current solution, when to retry with compact metacognitive feedback, and when to pass multiple attempts to a final aggregator. Across text, code, and multimodal reasoning benchmarks, our harness substantially improves a fixed Claude Sonnet-4.6 base model without parameter updates or benchmark-specific fine-tuning. On the evaluated public benchmark snapshots, it raises pooled accuracy from 48.3 to 56.9 and exceeds the strongest listed leaderboard entries on the three primary evaluation settings: HLE-Verified, LiveCodeBench v6, and R-Bench-V. These results suggest that strong LLMs may already possess useful metacognitive ability, but require an explicit control harness to act on it during reasoning.