MindShift: Analyzing Language Models' Reactions to Psychological Prompts

TL;DR

MindShift introduces a benchmark for evaluating how well large language models can adopt and reflect specified personality traits using psychometric measures, revealing variability across models and improvements over time.

Contribution

This paper presents MindShift, a novel benchmark for assessing LLMs' psychological adaptability through psychometric tests and detailed personality prompts.

Findings

LLMs show improved role perception with training advancements

Significant differences in responses across model types

LLMs can emulate human personality traits to varying degrees

Abstract

Large language models (LLMs) hold the potential to absorb and reflect personality traits and attitudes specified by users. In our study, we investigated this potential using robust psychometric measures. We adapted the most studied test in psychological literature, namely Minnesota Multiphasic Personality Inventory (MMPI) and examined LLMs' behavior to identify traits. To asses the sensitivity of LLMs' prompts and psychological biases we created personality-oriented prompts, crafting a detailed set of personas that vary in trait intensity. This enables us to measure how well LLMs follow these roles. Our study introduces MindShift, a benchmark for evaluating LLMs' psychological adaptability. The results highlight a consistent improvement in LLMs' role perception, attributed to advancements in training datasets and alignment techniques. Additionally, we observe significant differences in responses to psychometric assessments across different model types and families, suggesting variability in their ability to emulate human-like personality traits. MindShift prompts and code for LLM evaluation will be publicly available.