Annotating References to Mythological Entities in French Literature

TL;DR

This paper investigates the use of large language models, especially ChatGPT, for annotating and interpreting mythological references in French literature, highlighting their capabilities and limitations in accuracy and ethical concerns.

Contribution

It introduces an annotation scheme for mythological references and evaluates LLMs' effectiveness in annotation and interpretation tasks in literary analysis.

Findings

LLMs can effectively follow the annotation scheme with high accuracy.

ChatGPT provides valuable interpretative insights into mythological references.

LLMs often hallucinate and generate fabricated examples, raising ethical issues.

Abstract

In this paper, we explore the relevance of large language models (LLMs) for annotating references to Roman and Greek mythological entities in modern and contemporary French literature. We present an annotation scheme and demonstrate that recent LLMs can be directly applied to follow this scheme effectively, although not without occasionally making significant analytical errors. Additionally, we show that LLMs (and, more specifically, ChatGPT) are capable of offering interpretative insights into the use of mythological references by literary authors. However, we also find that LLMs struggle to accurately identify relevant passages in novels (when used as an information retrieval engine), often hallucinating and generating fabricated examples-an issue that raises significant ethical concerns. Nonetheless, when used carefully, LLMs remain valuable tools for performing annotations with high accuracy, especially for tasks that would be difficult to annotate comprehensively on a large scale through manual methods alone.