MapQaTor: An Extensible Framework for Efficient Annotation of Map-Based QA Datasets
Mahir Labib Dihan, Mohammed Eunus Ali, Md Rizwan Parvez

TL;DR
MapQaTor is an open-source framework that simplifies creating reliable, reproducible map-based QA datasets by integrating with map APIs, caching responses, and streamlining annotation and visualization, thus accelerating dataset development.
Contribution
It introduces a flexible, extensible platform for efficient map-based QA dataset creation, enabling seamless API integration and improved data reliability.
Findings
Speeds up annotation by at least 30 times compared to manual methods.
Facilitates evaluation of LLM-based geospatial reasoning.
Supports diverse map data sources and visualization.
Abstract
Mapping and navigation services like Google Maps, Apple Maps, OpenStreetMap, are essential for accessing various location-based data, yet they often struggle to handle natural language geospatial queries. Recent advancements in Large Language Models (LLMs) show promise in question answering (QA), but creating reliable geospatial QA datasets from map services remains challenging. We introduce MapQaTor, an extensible open-source framework that streamlines the creation of reproducible, traceable map-based QA datasets. MapQaTor enables seamless integration with any maps API, allowing users to gather and visualize data from diverse sources with minimal setup. By caching API responses, the platform ensures consistent ground truth, enhancing the reliability of the data even as real-world information evolves. MapQaTor centralizes data retrieval, annotation, and visualization within a single…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsData Management and Algorithms · Advanced Database Systems and Queries · Data Quality and Management
