Do LLMs Understand Why We Write Diaries? A Method for Purpose Extraction and Clustering

TL;DR

This paper presents a novel LLM-based method for extracting and clustering purposes behind diary writing, applied to Soviet-era diaries, revealing insights into authors' intentions and model limitations.

Contribution

Introduces a new LLM-driven approach for purpose extraction and clustering in diary analysis, evaluated on historical diaries with insights into model performance and errors.

Findings

GPT-4o and o1-mini outperform baseline models

Purpose analysis reveals gender, age, and temporal patterns

Models exhibit specific error types highlighting areas for improvement

Abstract

Diary analysis presents challenges, particularly in extracting meaningful information from large corpora, where traditional methods often fail to deliver satisfactory results. This study introduces a novel method based on Large Language Models (LLMs) to identify and cluster the various purposes of diary writing. By "purposes," we refer to the intentions behind diary writing, such as documenting life events, self-reflection, or practicing language skills. Our approach is applied to Soviet-era diaries (1922-1929) from the Prozhito digital archive, a rich collection of personal narratives. We evaluate different proprietary and open-source LLMs, finding that GPT-4o and o1-mini achieve the best performance, while a template-based baseline is significantly less effective. Additionally, we analyze the retrieved purposes based on gender, age of the authors, and the year of writing. Furthermore, we examine the types of errors made by the models, providing a deeper understanding of their limitations and potential areas for improvement in future research.