HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory

TL;DR

HyperLens offers a high-resolution method to analyze confidence trajectories in LLMs, revealing how task complexity affects cognitive effort and exposing effects of fine-tuning on inference dynamics.

Contribution

We introduce HyperLens, a novel tool for fine-grained analysis of confidence trajectories in LLMs, and demonstrate its effectiveness in quantifying cognitive effort during inference.

Findings

Complex tasks require higher cognitive effort in LLMs.

HyperLens reveals divergence in confidence trajectories between simple and complex tasks.

Fine-tuning can reduce cognitive effort and impair in-domain performance.

Abstract

While Large Language Models (LLMs) achieve strong performance across diverse tasks, their inference dynamics remain poorly understood because of the limited resolution of existing analysis tools. In this work, we identify an intrinsic magnification mechanism in transformer architectures: deeper layers inherently magnify the small changes of layer-wise confidence, providing a fine-grained confidence trajectory. Building on this insight, we introduce HyperLens, a high-resolution probe designed to trace confidence trajectories and quantify the cognitive effort during inference. Across LLMs and datasets, HyperLens reveals a consistent divergence in confidence trajectories that separates complex from simple tasks. We abstract this pattern into a quantitative cognitive effort metric. Our analysis reveals a fundamental principle: complex tasks consistently require higher cognitive effort. Finally, we provide a mechanistic diagnosis of a common side effect of standard Supervised Fine-Tuning (SFT): it can reduce cognitive effort and consequently degrade performance on in-domain tasks.