Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM

TL;DR

This study evaluates various NER models, including transformer-based architectures and LLMs, on Russian cultural news texts, revealing GPT-4o's superior performance with JSON prompts and highlighting rapid progress in model capabilities.

Contribution

It provides a comparative analysis of NER models on Russian cultural texts, introducing the SPbLitGuide dataset and demonstrating the effectiveness of GPT-4o with structured prompts.

Findings

GPT-4o achieves F1=0.93 with JSON prompts

GPT-4 has 0.99 precision in NER tasks

Follow-up GPT-4.1 evaluation yields F1=0.94

Abstract

This paper addresses the challenge of Named Entity Recognition (NER) for person names within the specialized domain of Russian news texts concerning cultural events. The study utilizes the unique SPbLitGuide dataset, a collection of event announcements from Saint Petersburg spanning 1999 to 2019. A comparative evaluation of diverse NER models is presented, encompassing established transformer-based architectures such as DeepPavlov, RoBERTa, and SpaCy, alongside recent Large Language Models (LLMs) including GPT-3.5, GPT-4, and GPT-4o. Key findings highlight the superior performance of GPT-4o when provided with specific prompting for JSON output, achieving an F1 score of 0.93. Furthermore, GPT-4 demonstrated the highest precision at 0.99. The research contributes to a deeper understanding of current NER model capabilities and limitations when applied to morphologically rich languages like Russian within the cultural heritage domain, offering insights for researchers and practitioners. Follow-up evaluation with GPT-4.1 (April 2025) achieves F1=0.94 for both simple and structured prompts, demonstrating rapid progress across model families and simplified deployment requirements.