REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

TL;DR

This paper introduces REMSA, an agent that automates the selection of remote sensing foundation models based on user queries, leveraging a new structured database and a benchmark of expert-verified scenarios to improve decision-making in diverse RS tasks.

Contribution

The paper presents the RSFM Database (RS-FMD) and REMSA, a novel constraint-aware agent that automates remote sensing foundation model selection using natural language queries and structured metadata.

Findings

REMSA outperforms baseline models in expert evaluations.

The system supports diverse RS tasks and data modalities.

Benchmark of 3,000 expert-verified configurations demonstrates REMSA's effectiveness.

Abstract

Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines. These models include unimodal vision encoders and multimodal architectures. FMs are adapted to diverse perception tasks, such as image classification, change detection, and visual question answering. However, selecting the most suitable remote sensing foundation model (RSFM) for a specific task remains challenging due to scattered documentation, heterogeneous formats, and complex deployment constraints. To address this, we first introduce the RSFM Database (RS-FMD), the first structured and schema-guided resource covering over 160 RSFMs trained on various data modalities, spanning different spatial, spectral, and temporal resolutions, considering different learning paradigms. Built upon RS-FMD, we further present REMSA, a constraint-aware agent that enables automated RSFM selection from natural language queries. REMSA combines structured FM metadata retrieval with a task-driven decision workflow. In detail, it interprets user input, clarifies missing constraints, ranks models via in-context learning, and provides transparent justifications. Our system supports various RS tasks and data modalities, enabling personalized, reproducible, and efficient FM selection. To evaluate REMSA, we construct a benchmark of 100 expert-verified RS query scenarios. Each query is evaluated across 4 systems and 3 LLM backbones, with the top-3 selected models manually assessed by domain experts. This results in 3,000 expert-scored task--system--model configurations under our novel expert-centered evaluation protocol. REMSA outperforms multiple baselines, showing its practical utility in real decision-making applications. REMSA operates entirely on publicly available metadata of open source RSFMs, without accessing private or sensitive data.