Can ChatGPT Assess Human Personalities? A General Evaluation Framework

TL;DR

This paper introduces a comprehensive framework for evaluating ChatGPT's ability to assess human personalities using MBTI tests, emphasizing unbiased prompts, flexible subject queries, and fairness metrics, revealing ChatGPT's potential and limitations.

Contribution

It presents a novel evaluation framework for assessing human personalities with LLMs, including unbiased prompting techniques and new metrics for consistency, robustness, and fairness.

Findings

ChatGPT can assess human personalities with reasonable consistency.

The framework improves fairness in personality assessments.

ChatGPT shows lower robustness to prompt biases compared to InstructGPT.

Abstract

Large Language Models (LLMs) especially ChatGPT have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. Existing works study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLMs. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. Specifically, we first devise unbiased prompts by randomly permuting options in MBTI questions and adopt the average testing result to encourage more impartial answer generation. Then, we propose to replace the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, we re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. The proposed framework enables LLMs to flexibly assess personalities of different groups of people. We further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and GPT-4. Our experiments reveal ChatGPT's ability to assess human personalities, and the average results demonstrate that it can achieve more consistent and fairer assessments in spite of lower robustness against prompt biases compared with InstructGPT.