Into the Unknown: Generating Geospatial Descriptions for New Environments

TL;DR

This paper introduces a large-scale data augmentation method using geospatial data and knowledge-graphs to improve the generation of spatial descriptions for new environments, significantly enhancing geolocation accuracy in unseen settings.

Contribution

The authors propose a novel augmentation approach combining CFG templates and LLMs to generate high-quality spatial instructions, improving geospatial reasoning in unfamiliar environments.

Findings

45.83% improvement in 100-meter accuracy on unseen environments

CFG-based augmentation outperforms LLM-based methods

Enhanced spatial reasoning with structured geospatial data

Abstract

Similar to vision-and-language navigation (VLN) tasks that focus on bridging the gap between vision and language for embodied navigation, the new Rendezvous (RVS) task requires reasoning over allocentric spatial relationships (independent of the observer's viewpoint) using non-sequential navigation instructions and maps. However, performance substantially drops in new environments with no training data. Using opensource descriptions paired with coordinates (e.g., Wikipedia) provides training data but suffers from limited spatially-oriented text resulting in low geolocation resolution. We propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. Our method constructs a grounded knowledge-graph, capturing entity relationships. Sampled entities and relations (`shop north of school') generate navigation instructions via (i) generating numerous templates using context-free grammar (CFG) to embed specific entities and relations; (ii) feeding the entities and relation into a large language model (LLM) for instruction generation. A comprehensive evaluation on RVS, showed that our approach improves the 100-meter accuracy by 45.83% on unseen environments. Furthermore, we demonstrate that models trained with CFG-based augmentation achieve superior performance compared with those trained with LLM-based augmentation, both in unseen and seen environments. These findings suggest that the potential advantages of explicitly structuring spatial information for text-based geospatial reasoning in previously unknown, can unlock data-scarce scenarios.