Can Large Language Models Generate Geospatial Code?

TL;DR

This paper evaluates the ability of large language models to generate geospatial code, introduces a comprehensive benchmark and framework, and demonstrates how domain-specific fine-tuning enhances performance.

Contribution

It presents GeoCode-Eval, a novel framework for assessing geospatial code generation, and introduces GeoCode-Bench, a large dataset for benchmarking LLMs in this domain.

Findings

Pre-training and instruction datasets improve geospatial code generation.

Fine-tuning on domain-specific data enhances model performance.

Evaluation reveals strengths and limitations of various LLMs in geospatial tasks.

Abstract

With the growing demand for spatiotemporal data processing and geospatial modeling, automating geospatial code generation has become essential for productivity. Large language models (LLMs) show promise in code generation but face challenges like domain-specific knowledge gaps and "coding hallucinations." This paper introduces GeoCode-Eval (GCE), a framework for assessing LLMs' ability to generate geospatial code across three dimensions: "Cognition and Memory," "Comprehension and Interpretation," and "Innovation and Creation," distributed across eight capability levels. We developed a benchmark dataset, GeoCode-Bench, consisting of 5,000 multiple-choice, 1,500 fill-in-the-blank, 1,500 true/false questions, and 1,000 subjective tasks covering code summarization, generation, completion, and correction. Using GeoCode-Bench, we evaluated three commercial closed-source LLMs, four open-source general-purpose LLMs, and 14 specialized code generation models. We also conducted experiments on few-shot and zero-shot learning, Chain of Thought reasoning, and multi-round majority voting to measure their impact on geospatial code generation. Additionally, we fine-tuned the Code LLaMA-7B model using Google Earth Engine-related JavaScript, creating GEECode-GPT, and evaluated it on subjective tasks. Results show that constructing pre-training and instruction datasets significantly improves code generation, offering insights for optimizing LLMs in specific domains.