Is ChatGPT a game changer for geocoding -- a benchmark for geocoding address parsing techniques

TL;DR

This paper evaluates GPT-3's performance in geocoding address parsing against traditional models using a newly created, diverse benchmark dataset of low-quality address inputs, highlighting current strengths and areas for improvement.

Contribution

It introduces a comprehensive benchmark dataset for geocoding address parsing and compares GPT-3 with transformer-based and LSTM models, providing insights into their relative performances.

Findings

Bidirectional LSTM-CRF outperforms GPT-3 and transformer models.

Transformer models perform comparably to LSTM-CRF.

GPT-3 shows potential with few-shot learning but needs further fine-tuning.

Abstract

The remarkable success of GPT models across various tasks, including toponymy recognition motivates us to assess the performance of the GPT-3 model in the geocoding address parsing task. To ensure that the evaluation more accurately mirrors performance in real-world scenarios with diverse user input qualities and resolve the pressing need for a 'gold standard' evaluation dataset for geocoding systems, we introduce a benchmark dataset of low-quality address descriptions synthesized based on human input patterns mining from actual input logs of a geocoding system in production. This dataset has 21 different input errors and variations; contains over 239,000 address records that are uniquely selected from streets across all U.S. 50 states and D.C.; and consists of three subsets to be used as training, validation, and testing sets. Building on this, we train and gauge the performance of the GPT-3 model in extracting address components, contrasting its performance with transformer-based and LSTM-based models. The evaluation results indicate that Bidirectional LSTM-CRF model has achieved the best performance over these transformer-based models and GPT-3 model. Transformer-based models demonstrate very comparable results compared to the Bidirectional LSTM-CRF model. The GPT-3 model, though trailing in performance, showcases potential in the address parsing task with few-shot examples, exhibiting room for improvement with additional fine-tuning. We open source the code and data of this presented benchmark so that researchers can utilize it for future model development or extend it to evaluate similar tasks, such as document geocoding.