Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report

TL;DR

This paper adapts clinical validity scales to evaluate the response validity of large language models using metacognitive probes, identifying invalid models and proposing a screening protocol.

Contribution

It introduces a validity scaling framework for LLMs, operationalizes six indices, and demonstrates their effectiveness in detecting invalid models.

Findings

Valid-profile models show significant confidence-item correlation.

Invalid-profile models lack confidence-item correlation.

Chain-of-thought training causes opposite response distortions.

Abstract

Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-llm