MarsRetrieval: Benchmarking Vision-Language Models for Planetary-Scale Geospatial Retrieval on Mars

TL;DR

MarsRetrieval introduces a comprehensive benchmark for evaluating vision-language models on Martian geospatial tasks, highlighting the challenges and importance of domain-specific fine-tuning for planetary exploration applications.

Contribution

It presents a new benchmark with multiple tasks for Mars geospatial discovery and evaluates various models, emphasizing the need for domain-specific adaptation.

Findings

Benchmark is challenging for existing models.

Domain-specific fine-tuning improves performance.

Multimodal models struggle with geomorphic distinctions.

Abstract

Data-driven approaches like deep learning are rapidly advancing planetary science, particularly in Mars exploration. Despite recent progress, most existing benchmarks remain confined to closed-set supervised visual tasks and do not support text-guided retrieval for geospatial discovery. We introduce MarsRetrieval, a retrieval benchmark for evaluating vision-language models for Martian geospatial discovery. MarsRetrieval includes three tasks: (1) paired image-text retrieval, (2) landform retrieval, and (3) global geo-localization, covering multiple spatial scales and diverse geomorphic origins. We propose a unified retrieval-centric protocol to benchmark multimodal embedding architectures, including contrastive dual-tower encoders and generative vision-language models. Our evaluation shows MarsRetrieval is challenging: even strong foundation models often fail to capture domain-specific geomorphic distinctions. We further show that domain-specific fine-tuning is critical for generalizable geospatial discovery in planetary settings. Our code is available at https://github.com/ml-stat-Sustech/MarsRetrieval