Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics

TL;DR

This paper investigates the stability of personality traits in large language models across languages, revealing inconsistencies that could impact AI safety, and proposes a new substrate-free psychometric framework.

Contribution

It introduces a novel substrate-free psychometric approach and demonstrates the instability of personality traits in language models across languages.

Findings

Language models show interlingual and intralingual personality instability.

Current models lack a consistent core personality, risking unsafe AI behavior.

Bayesian analysis reveals deeper-rooted issues in model personality traits.

Abstract

Previous research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.