Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance

TL;DR

This study systematically examines how the geographic and spectral diversity of pretraining datasets influence geospatial model performance, revealing spectral diversity as a key factor and providing new datasets and models.

Contribution

It introduces a comprehensive analysis of pretraining data diversity effects, highlighting spectral diversity's importance and releasing new datasets and models.

Findings

Pretraining on European data outperforms other datasets on global and local tasks.

Spectral diversity is strongly correlated with downstream performance.

Other diversity measures like continent, biome, landcover are weakly correlated.

Abstract

New geospatial foundation models introduce a new model architecture and pretraining dataset, often sampled using different notions of data diversity. Performance differences are largely attributed to the model architecture or input modalities, while the role of the pretraining dataset is rarely studied. To address this research gap, we conducted a systematic study on how the geographic composition of pretraining data affects a model's downstream performance. We created global and per-continent pretraining datasets and evaluated them on global and per-continent downstream datasets. We found that the pretraining dataset from Europe outperformed global and continent-specific pretraining datasets on both global and local downstream evaluations. To investigate the factors influencing a pretraining dataset's downstream performance, we analysed 10 pretraining datasets using diversity across continents, biomes, landcover and spectral values. We found that only spectral diversity was strongly correlated with performance, while others were weakly correlated. This finding establishes a new dimension of diversity to be accounted for when creating a high-performing pretraining dataset. We open-sourced 7 new pretraining datasets, pretrained models, and our experimental framework at https://github.com/kerner-lab/pretrain-where.