A Pragmatic Guide to Geoparsing Evaluation

TL;DR

This paper introduces a comprehensive evaluation framework for geoparsing, including a detailed taxonomy, standardized metrics, and a new dataset to improve consistency and real-world applicability of system assessments.

Contribution

It provides the first detailed taxonomy of toponyms, reviews evaluation metrics, and shares a new dataset to standardize and advance geoparsing evaluation practices.

Findings

Proposes a fine-grained toponym taxonomy for better task definition.

Recommends standardized metrics for geoparsing evaluation.

Shares GeoWebNews dataset for benchmarking and model development.

Abstract

Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real-world usage by the lack of distinction between the different types of toponyms, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of Toponyms. Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. Part 3) Evaluation Data: shared via a new dataset called GeoWebNews to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models.