Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain

TL;DR

This paper evaluates the performance of ten NER models on Dungeons and Dragons texts, revealing that some open-source models perform well without domain-specific modifications.

Contribution

It provides a comparative analysis of NER models in a specialized fantasy domain, highlighting the effectiveness of certain open-source models in this context.

Findings

Flair, Trankit, and Spacy outperform others in D&D NER tasks

Open-source LLMs can be effective for domain-specific NER with minimal adjustments

Domain-specific challenges impact NER performance in fantasy literature

Abstract

Many NLP tasks, although well-resolved for general English, face challenges in specific domains like fantasy literature. This is evident in Named Entity Recognition (NER), which detects and categorizes entities in text. We analyzed 10 NER models on 7 Dungeons and Dragons (D&D) adventure books to assess domain-specific performance. Using open-source Large Language Models, we annotated named entities in these books and evaluated each model's precision. Our findings indicate that, without modifications, Flair, Trankit, and Spacy outperform others in identifying named entities in the D&D context.