SATIN: A Multi-Task Metadataset for Classifying Satellite Imagery using Vision-Language Models

TL;DR

SATIN is a comprehensive multi-task metadataset derived from 27 satellite imagery datasets, designed to evaluate vision-language models' zero-shot classification across Earth's geographic diversity, highlighting current challenges and progress in remote sensing interpretation.

Contribution

This work introduces SATIN, the first large-scale, diverse satellite imagery metadataset, and evaluates vision-language models' zero-shot transfer capabilities on this challenging benchmark.

Findings

Strongest model achieves 52.0% accuracy

SATIN presents a challenging benchmark for remote sensing classification

Provides a public leaderboard to track model progress

Abstract

Interpreting remote sensing imagery enables numerous downstream applications ranging from land-use planning to deforestation monitoring. Robustly classifying this data is challenging due to the Earth's geographic diversity. While many distinct satellite and aerial image classification datasets exist, there is yet to be a benchmark curated that suitably covers this diversity. In this work, we introduce SATellite ImageNet (SATIN), a metadataset curated from 27 existing remotely sensed datasets, and comprehensively evaluate the zero-shot transfer classification capabilities of a broad range of vision-language (VL) models on SATIN. We find SATIN to be a challenging benchmark-the strongest method we evaluate achieves a classification accuracy of 52.0%. We provide a $\href{https://satinbenchmark.github.io}{\text{public leaderboard}}$ to guide and track the progress of VL models in this important domain.