Measuring Psychological Depth in Language Models

TL;DR

This paper introduces the Psychological Depth Scale (PDS), a new framework for evaluating the emotional and narrative complexity of stories generated by language models, validated through human and automated assessments.

Contribution

The paper presents the PDS framework rooted in literary theory, along with methods to automate it, enabling systematic evaluation of LLMs' storytelling depth from a reader's perspective.

Findings

Humans can reliably evaluate stories using PDS (Krippendorff's alpha 0.72)

GPT-4 with MoP achieves a 0.51 correlation with human judgments

GPT-4 stories match or surpass highly-rated human stories from Reddit

Abstract

Evaluations of creative stories generated by large language models (LLMs) often focus on objective properties of the text, such as its style, coherence, and diversity. While these metrics are indispensable, they do not speak to a story's subjective, psychological impact from a reader's perspective. We introduce the Psychological Depth Scale (PDS), a novel framework rooted in literary theory that measures an LLM's ability to produce authentic and narratively complex stories that provoke emotion, empathy, and engagement. We empirically validate our framework by showing that humans can consistently evaluate stories based on PDS (0.72 Krippendorff's alpha). We also explore techniques for automating the PDS to easily scale future analyses. GPT-4o, combined with a novel Mixture-of-Personas (MoP) prompting strategy, achieves an average Spearman correlation of 0.51 with human judgment while Llama-3-70B with constrained decoding scores as high as 0.68 for empathy. Finally, we compared the depth of stories authored by both humans and LLMs. Surprisingly, GPT-4 stories either surpassed or were statistically indistinguishable from highly-rated human-written stories sourced from Reddit. By shifting the focus from text to reader, the Psychological Depth Scale is a validated, automated, and systematic means of measuring the capacity of LLMs to connect with humans through the stories they tell.