Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures

TL;DR

This paper presents a scalable network analysis method to evaluate LLM-generated stories by examining social structures, revealing biases toward positive relationships, and aligning with human assessments.

Contribution

It introduces a novel network analysis approach for evaluating LLM storytelling, enabling large-scale, objective assessment of narrative social structures.

Findings

LLM stories show a bias toward positive, tightly-knit social relationships.

Network properties correlate with human judgments of story quality.

The method scales to over 1,200 stories for comprehensive analysis.

Abstract

Evaluating the creative capabilities of large language models (LLMs) in complex tasks often requires human assessments that are difficult to scale. We introduce a novel, scalable methodology for evaluating LLM story generation by analyzing underlying social structures in narratives as signed character networks. To demonstrate its effectiveness, we conduct a large-scale comparative analysis using networks from over 1,200 stories, generated by four leading LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, and Gemini 1.5 Flash) and a human-written corpus. Our findings, based on network properties like density, clustering, and signed edge weights, show that LLM-generated stories consistently exhibit a strong bias toward tightly-knit, positive relationships, which aligns with findings from prior research using human assessment. Our proposed approach provides a valuable tool for evaluating limitations and tendencies in the creative storytelling of current and future LLMs.