PhDGPT: Introducing a psychometric and linguistic dataset about how large language models perceive graduate students and professors in psychology

TL;DR

This paper introduces PhDGPT, a large dataset and framework capturing how GPT-3.5 perceives and simulates the psychometric and linguistic aspects of graduate students and professors in psychology, revealing both capabilities and limitations.

Contribution

It presents a novel dataset and prompting framework that model the machine psychology of academic figures, integrating psychometric scores with linguistic explanations for the first time.

Findings

LLMs can reconstruct human DASS factors with up to 80% accuracy.

Simulated academics show similar psychometric network structures to humans.

LLMs' explanations are less concrete and imageable for anxiety items.

Abstract

Machine psychology aims to reconstruct the mindset of Large Language Models (LLMs), i.e. how these artificial intelligences perceive and associate ideas. This work introduces PhDGPT, a prompting framework and synthetic dataset that encapsulates the machine psychology of PhD researchers and professors as perceived by OpenAI's GPT-3.5. The dataset consists of 756,000 datapoints, counting 300 iterations repeated across 15 academic events, 2 biological genders, 2 career levels and 42 unique item responses of the Depression, Anxiety, and Stress Scale (DASS-42). PhDGPT integrates these psychometric scores with their explanations in plain language. This synergy of scores and texts offers a dual, comprehensive perspective on the emotional well-being of simulated academics, e.g. male/female PhD students or professors. By combining network psychometrics and psycholinguistic dimensions, this study identifies several similarities and distinctions between human and LLM data. The psychometric networks of simulated male professors do not differ between physical and emotional anxiety subscales, unlike humans. Other LLMs' personification can reconstruct human DASS factors with a purity up to 80%. Furthemore, LLM-generated personifications across different scenarios are found to elicit explanations lower in concreteness and imageability in items coding for anxiety, in agreement with past studies about human psychology. Our findings indicate an advanced yet incomplete ability for LLMs to reproduce the complexity of human psychometric data, unveiling convenient advantages and limitations in using LLMs to replace human participants. PhDGPT also intriguingly capture the ability for LLMs to adapt and change language patterns according to prompted mental distress contextual features, opening new quantitative opportunities for assessing the machine psychology of these artificial intelligences.