PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection

TL;DR

PsyCoT leverages psychological questionnaires as structured chain-of-thought processes to significantly enhance GPT-3.5's ability to detect personality traits from text, demonstrating improved accuracy and robustness.

Contribution

This paper introduces PsyCoT, a novel multi-turn dialogue approach that mimics psychological questionnaires to improve personality detection with LLMs.

Findings

PsyCoT improves GPT-3.5's F1 score by 4.23/10.63 points on benchmark datasets.

The method enhances the robustness of personality detection in LLMs.

Incorporating structured CoT processes boosts inference quality in personality analysis.

Abstract

Recent advances in large language models (LLMs), such as ChatGPT, have showcased remarkable zero-shot performance across various NLP tasks. However, the potential of LLMs in personality detection, which involves identifying an individual's personality from their written texts, remains largely unexplored. Drawing inspiration from Psychological Questionnaires, which are carefully designed by psychologists to evaluate individual personality traits through a series of targeted items, we argue that these items can be regarded as a collection of well-structured chain-of-thought (CoT) processes. By incorporating these processes, LLMs can enhance their capabilities to make more reasonable inferences on personality from textual input. In light of this, we propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner. In particular, we employ a LLM as an AI assistant with a specialization in text analysis. We prompt the assistant to rate individual items at each turn and leverage the historical rating results to derive a conclusive personality preference. Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection, achieving an average F1 score improvement of 4.23/10.63 points on two benchmark datasets compared to the standard prompting method. Our code is available at https://github.com/TaoYang225/PsyCoT.