"Does the cafe entrance look accessible? Where is the door?" Towards Geospatial AI Agents for Visual Inquiries

TL;DR

This paper proposes Geo-Visual Agents, multimodal AI systems that analyze geospatial images and GIS data to answer complex visual-spatial questions about the world, extending beyond traditional map data.

Contribution

It introduces the concept of Geo-Visual Agents capable of understanding and responding to nuanced visual inquiries using large-scale geospatial image repositories.

Findings

Conceptual framework for Geo-Visual Agents

Three exemplars demonstrating potential applications

Identification of key challenges and future opportunities

Abstract

Interactive digital maps have revolutionized how people travel and learn about the world; however, they rely on pre-existing structured data in GIS databases (e.g., road networks, POI indices), limiting their ability to address geo-visual questions related to what the world looks like. We introduce our vision for Geo-Visual Agents--multimodal AI agents capable of understanding and responding to nuanced visual-spatial inquiries about the world by analyzing large-scale repositories of geospatial images, including streetscapes (e.g., Google Street View), place-based photos (e.g., TripAdvisor, Yelp), and aerial imagery (e.g., satellite photos) combined with traditional GIS data sources. We define our vision, describe sensing and interaction approaches, provide three exemplars, and enumerate key challenges and opportunities for future work.