NeSy-Route: A Neuro-Symbolic Benchmark for Constrained Route Planning in Remote Sensing

TL;DR

NeSy-Route is a large-scale neuro-symbolic benchmark designed to evaluate and improve the planning capabilities of multimodal large language models in remote sensing applications, addressing current evaluation limitations.

Contribution

It introduces an automated data-generation framework and a hierarchical evaluation protocol for comprehensive assessment of perception, reasoning, and planning in remote sensing tasks.

Findings

Existing MLLMs have significant deficiencies in perception and planning.

NeSy-Route includes over 10,000 diverse route-planning samples with optimal solutions.

The benchmark enables detailed analysis of model capabilities across multiple levels.

Abstract

Remote sensing underpins crucial applications such as disaster relief and ecological field surveys, where systems must understand complex scenes and constraints and make reliable decisions. Current remote-sensing benchmarks mainly focus on evaluating perception and reasoning capabilities of multimodal large language models (MLLMs). They fail to assess planning capability, stemming either from the difficulty of curating and validating planning tasks at scale or from evaluation protocols that are inaccurate and inadequate. To address these limitations, we introduce NeSy-Route, a large-scale neuro-symbolic benchmark for constrained route planning in remote sensing. Within this benchmark, we introduce an automated data-generation framework that integrates high-fidelity semantic masks with heuristic search to produce diverse route-planning tasks with provably optimal solutions. This allows NeSy-Route to comprehensively evaluate planning across 10,821 route-planning samples, nearly 10 times larger than the largest prior benchmark. Furthermore, a three-level hierarchical neuro-symbolic evaluation protocol is developed to enable accurate assessment and support fine-grained analysis on perception, reasoning, and planning simultaneously. Our comprehensive evaluation of various state-of-the-art MLLMs demonstrates that existing MLLMs show significant deficiencies in perception and planning capabilities. We hope NeSy-Route can support further research and development of more powerful MLLMs for remote sensing.