Leveraging Implicit Sentiments: Enhancing Reliability and Validity in Psychological Trait Evaluation of LLMs

TL;DR

This paper introduces the Core Sentiment Inventory (CSI), a bilingual tool for evaluating LLMs' psychological traits, significantly improving reliability and validity over existing methods by capturing nuanced emotional tendencies across languages.

Contribution

The paper presents a novel bilingual evaluation instrument, CSI, specifically designed for LLMs, enhancing psychological assessment accuracy and consistency compared to traditional approaches.

Findings

CSI effectively captures emotional patterns in LLMs.

CSI shows high reliability and consistency in results.

CSI scores strongly correlate (>0.85) with real-world sentiment outputs.

Abstract

Recent advancements in Large Language Models (LLMs) have led to their increasing integration into human life. With the transition from mere tools to human-like assistants, understanding their psychological aspects-such as emotional tendencies and personalities-becomes essential for ensuring their trustworthiness. However, current psychological evaluations of LLMs, often based on human psychological assessments like the BFI, face significant limitations. The results from these approaches often lack reliability and have limited validity when predicting LLM behavior in real-world scenarios. In this work, we introduce a novel evaluation instrument specifically designed for LLMs, called Core Sentiment Inventory (CSI). CSI is a bilingual tool, covering both English and Chinese, that implicitly evaluates models' sentiment tendencies, providing an insightful psychological portrait of LLM across three dimensions: optimism, pessimism, and neutrality. Through extensive experiments, we demonstrate that: 1) CSI effectively captures nuanced emotional patterns, revealing significant variation in LLMs across languages and contexts; 2) Compared to current approaches, CSI significantly improves reliability, yielding more consistent results; and 3) The correlation between CSI scores and the sentiment of LLM's real-world outputs exceeds 0.85, demonstrating its strong validity in predicting LLM behavior. We make CSI public available via: https://github.com/dependentsign/CSI.