MapQaTor: An Extensible Framework for Efficient Annotation of Map-Based QA Datasets

TL;DR

MapQaTor is an open-source framework that simplifies creating reliable, reproducible map-based QA datasets by integrating with map APIs, caching responses, and streamlining annotation and visualization, thus accelerating dataset development.

Contribution

It introduces a flexible, extensible platform for efficient map-based QA dataset creation, enabling seamless API integration and improved data reliability.

Findings

Speeds up annotation by at least 30 times compared to manual methods.

Facilitates evaluation of LLM-based geospatial reasoning.

Supports diverse map data sources and visualization.

Abstract

Mapping and navigation services like Google Maps, Apple Maps, OpenStreetMap, are essential for accessing various location-based data, yet they often struggle to handle natural language geospatial queries. Recent advancements in Large Language Models (LLMs) show promise in question answering (QA), but creating reliable geospatial QA datasets from map services remains challenging. We introduce MapQaTor, an extensible open-source framework that streamlines the creation of reproducible, traceable map-based QA datasets. MapQaTor enables seamless integration with any maps API, allowing users to gather and visualize data from diverse sources with minimal setup. By caching API responses, the platform ensures consistent ground truth, enhancing the reliability of the data even as real-world information evolves. MapQaTor centralizes data retrieval, annotation, and visualization within a single platform, offering a unique opportunity to evaluate the current state of LLM-based geospatial reasoning while advancing their capabilities for improved geospatial understanding. Evaluation metrics show that, MapQaTor speeds up the annotation process by at least 30 times compared to manual methods, underscoring its potential for developing geospatial resources, such as complex map reasoning datasets. The website is live at: https://mapqator.github.io/ and a demo video is available at: https://youtu.be/bVv7-NYRsTw.