GPT-4 Generated Narratives of Life Events using a Structured Narrative Prompt: A Validation Study

TL;DR

This study evaluates GPT-4's ability to generate and classify life event narratives using structured prompts, demonstrating high validity in generated narratives and developing ML models for automated classification.

Contribution

It introduces a structured prompt approach for narrative generation and trains ML models to automate validity assessment of LLM-generated narratives.

Findings

87.43% of narratives conveyed intended events

ML models effectively identified valid narratives

Challenges remain in classifying invalid narratives

Abstract

Large Language Models (LLMs) play a pivotal role in generating vast arrays of narratives, facilitating a systematic exploration of their effectiveness for communicating life events in narrative form. In this study, we employ a zero-shot structured narrative prompt to generate 24,000 narratives using OpenAI's GPT-4. From this dataset, we manually classify 2,880 narratives and evaluate their validity in conveying birth, death, hiring, and firing events. Remarkably, 87.43% of the narratives sufficiently convey the intention of the structured prompt. To automate the identification of valid and invalid narratives, we train and validate nine Machine Learning models on the classified datasets. Leveraging these models, we extend our analysis to predict the classifications of the remaining 21,120 narratives. All the ML models excelled at classifying valid narratives as valid, but experienced challenges at simultaneously classifying invalid narratives as invalid. Our findings not only advance the study of LLM capabilities, limitations, and validity but also offer practical insights for narrative generation and natural language processing applications.