Automated Motif Indexing on the Arabian Nights

TL;DR

This paper introduces the first computational method for motif indexing in folklore texts, using the Arabian Nights and a detailed motif index, achieving high accuracy with fine-tuned language models.

Contribution

It presents a novel automated approach to motif indexing, leveraging a large annotated corpus and various NLP techniques, including fine-tuned LLMs, to improve motif detection.

Findings

Fine-tuned Llama3 achieves 0.85 F1 score.

Multiple approaches tested, with fine-tuned LLMs performing best.

Annotated corpus of 58,450 sentences created for training and testing.

Abstract

Motifs are non-commonplace, recurring narrative elements, often found originally in folk stories. In addition to being of interest to folklorists, motifs appear as metaphoric devices in modern news, literature, propaganda, and other cultural texts. Finding expressions of motifs in the original folkloristic text is useful for both folkloristic analysis (motif indexing) as well as for understanding the modern usage of motifs (motif detection and interpretation). Prior work has primarily shown how difficult these problems are to tackle using automated techniques. We present the first computational approach to motif indexing. Our choice of data is a key enabler: we use a large, widely available text (the Arabian Nights) paired with a detailed motif index (by El-Shamy in 2006), which overcomes the common problem of inaccessibility of texts referred to by the index. We created a manually annotated corpus that identified 2,670 motif expressions of 200 different motifs across 58,450 sentences for training and testing. We tested five types of approaches for detecting motif expressions given a motif index entry: (1) classic retrieve and re-rank using keywords and a fine-tuned cross-encoder; (2) off-the-shelf embedding models; (3) fine-tuned embedding models; (4) generative prompting of off-the-shelf LLMs in N-shot setups; and (5) the same generative approaches on LLMs fine-tuned with LoRA. Our best performing system is a fine-tuned Llama3 model which achieves an overall performance of 0.85 F1.