Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection

TL;DR

This study evaluates and compares the performance of advanced NLP models and large language models in multilingual geo-entity detection, highlighting their strengths and challenges across different languages and datasets.

Contribution

It provides a comprehensive benchmarking of NLP and LLMs for multilingual geo-entity detection, informing future development of more effective geospatial NLP tools.

Findings

GPT-4 outperforms other models in accuracy and F1 scores.

Multilingual models show varied effectiveness across languages.

Challenges remain in achieving high precision in Arabic and Russian datasets.

Abstract

The integration of advanced Natural Language Processing (NLP) methodologies and Large Language Models (LLMs) has significantly enhanced the extraction and analysis of geospatial data from multilingual texts, impacting sectors such as national and international security. This paper presents a comprehensive evaluation of leading NLP models -- SpaCy, XLM-RoBERTa, mLUKE, GeoLM -- and LLMs, specifically OpenAI's GPT 3.5 and GPT 4, within the context of multilingual geo-entity detection. Utilizing datasets from Telegram channels in English, Russian, and Arabic, we examine the performance of these models through metrics such as accuracy, precision, recall, and F1 scores, to assess their effectiveness in accurately identifying geospatial references. The analysis exposes each model's distinct advantages and challenges, underscoring the complexities involved in achieving precise geo-entity identification across varied linguistic landscapes. The conclusions drawn from this experiment aim to direct the enhancement and creation of more advanced and inclusive NLP tools, thus advancing the field of geospatial analysis and its application to global security.