Machine individuality: Separating genuine idiosyncrasy from response bias in large language models

TL;DR

This study introduces a method to distinguish genuine individual differences in large language models from response biases, revealing that a significant portion of their behavior is due to stable, stimulus-specific traits.

Contribution

The paper applies psychometric models to large language models to quantify and characterize their unique, stable behavioral fingerprints, advancing understanding of LLM individuality.

Findings

16.9% of variance is due to stimulus-specific individuality

Individual differences form a coherent, model-specific fingerprint

Genuine individuality exceeds response biases and noise

Abstract

As large language models (LLMs) are increasingly integrated into daily life, in roles ranging from high-stakes decision support to companionship, understanding their behavioral dispositions becomes critical. A growing literature uses psychometric inventories and cognitive paradigms to profile LLM dispositions. However, these approaches cannot determine whether behavioral differences reflect stable, stimulus-specific individuality or global response biases and stochastic noise. Here, we apply crossed random-effects models -- widely used in psychometrics to separate systematic effects -- to 74.9 million ratings provided by 10 open-weight LLMs for over 100,000 words across 14 psycholinguistic norms. On average, 16.9% of variance is attributable to stimulus-specific individuality, robustly exceeding a statistical null model. Cross-norm prediction analyses reveal this individuality as a coherent fingerprint, unique to each model. These results identify individual differences among LLMs that cannot be attributed to response biases or stochastic noise. We term these differences machine individuality.